OpenVPN failing: ERROR: Cannot open TUN/TAP dev /dev/net/tun: Operation not permitted (errno=1)

[jata1]

First of all - I wanted to say thanks for such a great image/project. Works really well.

This morning I restarted my server and now the container does not come up properly. From what I can see, everything looks fine/normal but I get this error and the container just tries to restart.

I have not changed anything on my system - the only thing I remember is that container.io package was updated recently.

Not sure if this is VPN provider related or docker or the container. log below (redacted a few bits)

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-custom-files: starting
[custom-init] No custom files found, skipping...
s6-rc: info: service init-custom-files successfully started
s6-rc: info: service init-environment: starting
Image build from commit 7b9d766 on 2024-11-25 15:45:53
--------------------
2024-11-30 14:57:32 [DEBUG] Docker interface defined as eth0
2024-11-30 14:57:32 [DEBUG] Docker IPv4 address defined as 172.3.0.12
2024-11-30 14:57:32 [INFO] Docker IPv4 network defined as 172.3.0.0/16
2024-11-30 14:57:32 [DEBUG] Default IPv4 gateway defined as 172.3.0.1
2024-11-30 14:57:32 [INFO] PUID defined as 1001
2024-11-30 14:57:32 [INFO] PGID defined as 100
2024-11-30 14:57:32 [DEBUG] A user with PUID 1001 already exists in /etc/passwd, nothing to do.
2024-11-30 14:57:32 [INFO] VPN_ENABLED defined as 'yes'
2024-11-30 14:57:32 [INFO] VPN_TYPE defined as 'openvpn'
2024-11-30 14:57:32 [INFO] NAME_SERVERS defined as '8.8.8.8,8.8.4.4'
2024-11-30 14:57:32 [DEBUG] Adding 8.8.8.8 to resolv.conf
2024-11-30 14:57:32 [DEBUG] Adding 8.8.4.4 to resolv.conf
s6-rc: info: service init-environment successfully started
s6-rc: info: service init-vpn: starting
2024-11-30 14:57:32 [INFO] Choosen VPN config: 'pia_sydney2.ovpn'
2024-11-30 14:57:32 [INFO] Using credentials from /config/openvpn/pia_sydney2_credentials.conf
2024-11-30 14:57:32 [INFO] VPN remote line defined as 'au-sydney.privacy.network 1197'
2024-11-30 14:57:32 [INFO] VPN_REMOTE defined as 'au-sydney.privacy.network'
2024-11-30 14:57:32 [INFO] VPN_PORT defined as '1197'
2024-11-30 14:57:32 [INFO] VPN_PROTOCOL defined as 'udp'
2024-11-30 14:57:32 [INFO] VPN_DEVICE_TYPE defined as 'tun0'
2024-11-30 14:57:32 [DEBUG] Route: 8.8.8.8 via 172.3.0.1 dev eth0 src 172.3.0.12 uid 0 
2024-11-30 14:57:32 [DEBUG] Ping to 8.8.8.8 succeeded
2024-11-30 14:57:32 [DEBUG] au-sydney.privacy.network resolved to 191.101.210.208
2024-11-30 14:57:32 [DEBUG] Ping to 191.101.210.208 via eth0 succeeded
2024-11-30 14:57:32 [INFO] Starting OpenVPN...
--------------------
2024-11-30 14:57:32 [DEBUG] OpenVPN PID: 169
2024-11-30 14:57:32 DEPRECATED OPTION: --cipher set to 'aes-256-cbc' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 
2024-11-30 14:57:32 WARNING: file '/config/openvpn/pia_sydney2_credentials.conf' is group or others accessible
2024-11-30 14:57:32 OpenVPN 2.6.11 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-11-30 14:57:32 library versions: OpenSSL 3.1.7 3 Sep 2024, LZO 2.10
2024-11-30 14:57:32 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-11-30 14:57:32 CRL: loaded 1 CRLs from file -----BEGIN X509 CRL-----[redacted]-----END X509 CRL-----
2024-11-30 14:57:32 TCP/UDP: Preserving recently used remote address: [AF_INET]117.120.9.22:1197
2024-11-30 14:57:32 UDPv4 link local: (not bound)
2024-11-30 14:57:32 UDPv4 link remote: [AF_INET]117.120.9.22:1197
2024-11-30 14:57:32 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2024-11-30 14:57:32 [sydney420] Peer Connection Initiated with [AF_INET]117.120.9.22:1197
2024-11-30 14:57:32 sitnl_send: rtnl: generic error (-101): Network unreachable
2024-11-30 14:57:32 ERROR: Cannot open TUN/TAP dev /dev/net/tun: Operation not permitted (errno=1)
2024-11-30 14:57:32 Exiting due to fatal error
--------------------
2024-11-30 14:57:32 [ERROR] Failed to start OpenVPN
s6-rc: warning: unable to start service init-vpn: command exited 1
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
/run/s6/basedir/scripts/rc.init: fatal: stopping the container.
s6-rc: info: service init-environment: stopping
s6-rc: info: service init-environment successfully stopped
s6-rc: info: service init-custom-files: stopping
s6-rc: info: service init-custom-files successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

[jata1]

if helpful, this is my compose but as I said this has just suddenly stopped working.

So must be docker or maybe issue with vpn provider (PIA)

services:
  qbittorrent:
    image: trigus42/qbittorrentvpn:latest
    container_name: qbittorrent
    hostname: qbittorrent
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    environment:
      - DEBUG=yes
      - HEALTH_CHECK_HOST=8.8.8.8
      - HEALTH_CHECK_INTERVAL=10
      - HEALTH_CHECK_TIMEOUT=30
      - DOWNLOAD_DIR_CHOWN=yes
      - VPN_ENABLED=yes
      - VPN_TYPE=openvpn
      - NAME_SERVERS=8.8.8.8,8.8.4.4
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - UMASK=002
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /symlinks/omv-system/appdata/qbittorrent/config:/config
      - /symlinks/media/torrent:/media/torrent # SKIP_BACKUP
      - /symlinks/media/torrent:/downloads # SKIP_BACKUP
    ports:
      - 9092:8080
    restart: unless-stopped
    networks:
      minipc-bridge:

networks:
  minipc-bridge:
    name: minipc-bridge
    external: true   

[Trigus42]

ERROR: Cannot open TUN/TAP dev /dev/net/tun: Operation not permitted (errno=1) may indicate that the TUN module is missing. Restarting your server might have applied a new kernel update. Please try running modprobe tun on the host and see if that works and if it fixes your issue.

If it does, to permanently solve this, you might want to add the SYS_MODULE capability to your container. Additionally, you may need to add the volume /lib/modules:/lib/modules:ro.

Another thing you can try is adding the following to your compose file:

devices:
  - /dev/net/tun

I'm not sure what causes sitnl_send: rtnl: generic error (-101): Network unreachable, but maybe it's just a side effect of the other error.

[jata1]

Thanks for the prompt and helpful response.

I tried modprobe but this did not resolve the issue on either my prod (amd64 minipc) and test (arm64 rpi5) docker setups running latest debian 12 kernel or proxmox 6.8 kernel

Adding the device (/dev/net/tun) to my compose file has resolved one error in the log and the container starts fine now in prod and test.

I still get the error sitnl_send: rtnl: generic error (-101): Network unreachable and I don't recall seeing this previously.

Very strange that it worked fine before without the device mapping in compose.

I'll let you know if I can find out what is causing the network unreachable error.

[jata1]

do you think the network unreachable error could be ipv6 related as i am using ipv4?

[Trigus42]

Please try increasing your OpenVPN logging verbosity by adding verb 4 or higher to your OVPN config until you get some useful information about the error. Please note that logs above a certain level (I believe it’s above level 6, but I'm not entirely sure) may contain sensitive information.

do you think the network unreachable error could be ipv6 related as i am using ipv4?

Unlikely as IPv6 config options should be filtered out if you don't have IPv6 connectivity:

alpine-qbittorrentvpn/rootfs/scripts/init-scripts/vpn.sh

Lines 221 to 226 in 7b9d766

	openvpn \
	--pull-filter ignore "route-ipv6" --pull-filter ignore "ifconfig-ipv6" --pull-filter ignore "tun-ipv6" --pull-filter ignore "redirect-gateway ipv6" --pull-filter ignore "dhcp-option DNS6" \
	--auth-user-pass /config/openvpn/"${VPN_CONFIG_NAME}"_credentials.conf \
	--config "${VPN_CONFIG}" \
	--script-security 2 \
	--route-up /scripts/helper/resume-after-connect.sh \

Maybe you need to explicitly disable IPv6 in your container:

sysctls:
  - net.ipv6.conf.all.disable_ipv6=1

---

If you want to share sensitive information with me, you can use this PGP key for encryption:

Trigus42_0xD14B5AEB_public.asc

-----BEGIN PGP PUBLIC KEY BLOCK-----

mDMEZ0wyphYJKwYBBAHaRw8BAQdAMpV/BGbt2wCPLxxYR6tribKWlSV5n8HmmBdK
CVQmEKG0CFRyaWd1czQyiJkEExYKAEEWIQTMjgh1UnH8w2q7H00efh/B0Uta6wUC
Z0wypgIbAwUJBaOpigULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIXgAAKCRAefh/B
0Uta6xHdAQCkdcX/losJTrZH1GRUqayGr/UjDn/+zOG+WrMZm64CgwEA1dFxi1ED
7nxV4QnrxYK326aaLrokjAlUjkAZM5+GYgu4OARnTDKmEgorBgEEAZdVAQUBAQdA
/FWUlCUZAlR/0OnsKXcyoRjy2FULDH68L3LREJdHe1cDAQgHiH4EGBYKACYWIQTM
jgh1UnH8w2q7H00efh/B0Uta6wUCZ0wypgIbDAUJBaOpigAKCRAefh/B0Uta6wf/
AQCDpfnL5Un81a/6e3cVWxd/Pq294ncP/bNR1puH62SR0gD/QY7WHy0FbSI0F/ZD
PXvVGPpa09JRk/EeLmbYumNWFws=
=cno7
-----END PGP PUBLIC KEY BLOCK-----  

[jata1]

Thanks again for your help.

Looks like this issue is related to the latest version of containerd.io - see qdm12/gluetun#2606

I tried disabling ipv6 but this made no difference. I'll let you know if I track anything down by increasing the openvpn logging

sysctls:
  - net.ipv6.conf.all.disable_ipv6=1

[acrobatmaxx]

Hello guys,

This morning I also upgraded containerd.io without really thinking about it, and it created the same issue as @jata1. I did the steps Trigus42 suggested. Running modprobe tun didn't do anything, but adding the device (dev.net.tun) to my compose completely fixed the issue for me.

I'm running Ubuntu 22.04.05 LTS 6.8 Kernel

I use PrivateVPN

Let me know if any of my logs and or any additional info I could provide would help with the issue of OP.

Thanks!

[jata1]

I think this is an intentional change upstream in containerd.io and not expecting it to change back.

containerd/containerd#11078

Might require that the instructions/guide for this (and other) containers will need to be updated.

[V0l-D]

This issue also started for me as @acrobatmaxx mentioned adding that specific device fixes the issue here's a compose file as example:

version: "3.3"

services:
  qbittorrent:
    image: trigus42/qbittorrentvpn
    container_name: qbittorrent
    environment:
      - VPN_TYPE=openvpn
      - VPN_USERNAME=MY_VPN_USERNAME
      - VPN_PASSWORD=MY_VPN_PASSWORD
      - WEBUI_PASSWORD=MY_WEBUI_PASSWORD
    volumes:
      - './config/:/config'
    ports:
      - 80:8080
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    devices:
      - /dev/net/tun:/dev/net/tun

[Trigus42]

@jata1 Since the main issue has been resolved, I’ve closed this thread. If you continue to have the sitnl_send: rtnl: generic error (-101): Network unreachable problem, please feel free to open a new issue

[mafosa]

Hi there,
running Debian 12, with containerd.io v1.7.24-1 docker can access /dev/net/tun in standalone mode with these options in compose file: privileged: true and cap_add: NET_ADMIN. No need to set devices: /dev/net/tun.
But when running within a stack in docker swarm, I must downgrade to container.io v1.7.23-1
No way in v1.7.24-1 so I cant upgrade the whole system.