Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rancher/k3s:v1.31.3-k3s1 messing with network device permissions #11429

Closed
robjackstewart opened this issue Dec 8, 2024 · 12 comments
Closed

Comments

@robjackstewart
Copy link

Environmental Info:
K3s Version:
1.31.3-k3s1

Node(s) CPU architecture, OS, and Version:
Linux k3d-home-media-server-server-0 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 GNU/Linux

Cluster Configuration:

# k3d-config.yaml
apiVersion: k3d.io/v1alpha5
kind: Simple 
metadata:
  name: k3s-1.31.3-test
servers: 1
agents: 0
image: rancher/k3s:v1.31.2-k3s1 # changing to rancher/k3s:v1.31.3-k3s1 causes aforementioned error
network: test-network

Describe the bug:
Upgrading the rancher image from rancher/k3s:v1.31.2-k3s1 to rancher/k3s:v1.31.3-k3s1 causes permissions errors when accessing /dev/net/tun inside the gluetun container.

Please find the file structure and contents as follows.

File structure:

- k3d-config.broken.yaml
- k3d-config.working.yaml
- templates/
  - gluetun.yaml
- Chart.yaml

File contents:

# k3d-config.broken.yaml
apiVersion: k3d.io/v1alpha5
kind: Simple 
metadata:
  name: k3s-1.31.3-broken
servers: 1
agents: 0
image: rancher/k3s:v1.31.3-k3s1
network: test-network
# k3d-config.working.yaml
apiVersion: k3d.io/v1alpha5
kind: Simple 
metadata:
  name: k3s-1.31.3-working
servers: 1
agents: 0
image: rancher/k3s:v1.31.2-k3s1
network: test-network
# Chart.yaml
apiVersion: v2
name: k3s-1.31.3-test
version: 0.0.1
appVersion: v0.0.1
# templates/gluetun.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gluetun
spec:
  selector:
    matchLabels:
      app: transmission
  template:
    metadata:
      labels:
        app: transmission
    spec:
      containers:
        - name: gluetun
          image: ghcr.io/qdm12/gluetun:latest
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          env:
            - name: VPN_SERVICE_PROVIDER
              value: mullvad
            - name: VPN_TYPE
              value: openvpn
            - name: OPENVPN_USER
              value: # redacted
            - name: OPENVPN_PASSWORD
              value: m
            - name: SERVER_COUNTRIES
              value: UK
            - name: OWNED_ONLY
              value: 'yes'

Steps To Reproduce:

  1. Clone this repo for reproduction
  2. Populate appropriate mullvad username as the value of the OPENVPN_USER environment variable in templates/gluetun.yaml
  3. Run k3d cluster create --config k3d-config.working.yaml
  4. Run helm upgrade --install k3s-1.31.3-test .
  5. Wait for the glueton-0 pod to Running
  6. Run kubectl logs gluetun-0 and note successfully running
  7. Run k3d cluster create --config k3d-config.broken.yaml
  8. Run helm upgrade --install k3s-1.31.3-test .
  9. Wait for the glueton-0 pod to Error
  10. Run kubectl logs gluetun-0 and note permission denied error for /dev/net/tun

Expected behavior:
The /dev/net/tun file should be accessible as it was in v1.31.2-k3s1.

Actual behavior:
Getting the a permission denied error inside the gluetun container when attempting to open the /dev/net/tun device.

Additional context / logs:

Error from inside gluetun container when using v1.31.3-k3s1

2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO [routing] adding route for 0.0.0.0/0
2024-12-08T12:51:29Z INFO [firewall] setting allowed subnets...
2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
2024-12-08T12:51:29Z INFO [routing] routing cleanup...
2024-12-08T12:51:29Z INFO [routing] default route found: interface eth0, gateway 10.42.0.1, assigned IP 10.42.0.15 and family v4
2024-12-08T12:51:29Z INFO [routing] deleting route for 0.0.0.0/0
2024-12-08T12:51:29Z ERROR creating tun device: unix opening TUN device file: operation not permitted (did you specify --device /dev/net/tun to your container command?)
2024-12-08T12:51:29Z INFO Shutdown successful
@robjackstewart robjackstewart changed the title rancher/k3s:1.31.3-k3s1 messing with network device permissions rancher/k3s:v1.31.3-k3s1 messing with network device permissions Dec 8, 2024
@brandond
Copy link
Member

brandond commented Dec 9, 2024

Can you reproduce this on standalone k3s or is this only a k3d problem? I have no idea what you're doing with this here.

@maggie44
Copy link

maggie44 commented Dec 11, 2024

Cross linking issues here. Not sure where the source of the issue is: kubernetes/kubernetes#129157

I am using the stable tag and the version bumped last night.

@brandond
Copy link
Member

brandond commented Dec 11, 2024

Possibly a containerd or runc change? What specific versions of k3s did you upgrade from/to? Are you using selinux?

@maggie44
Copy link

maggie44 commented Dec 11, 2024

Possibly a containerd or runc change? What specific versions of k3s did you upgrade from/to? Are you using selinux?

I am using the stable channel, so I assume this: c827805

1.30.6 -> 1.31.3

And yes, selinux.

@maggie44
Copy link

Following this thread: opencontainers/runc#3468 (comment)

For docker, seems like adding:

devices:
      - /dev/net/tun:/dev/net/tun

The equivalent in Kubernetes though still seems to required privileged permissions:

	// Define the volume and volume mount for /dev/net/tun
	tunDeviceVolume := corev1.Volume{
		Name: "tun-device",
		VolumeSource: corev1.VolumeSource{
			HostPath: &corev1.HostPathVolumeSource{
				Path: "/dev/net/tun",
				Type: func() *corev1.HostPathType {
					charDevice := corev1.HostPathCharDev
					return &charDevice
				}(),
			},
		},
	}

	tunDeviceVolumeMount := corev1.VolumeMount{
		Name:      "tun-device",
		MountPath: "/dev/net/tun",
		ReadOnly:  false,

@maggie44
Copy link

Traced back to runc, an intended break. Unfortunately means granting privileged now for creating tun devices, and during the transition containers will fail to start: tailscale/tailscale#14262 (comment)

@brandond
Copy link
Member

brandond commented Dec 11, 2024

Closing, as this was an intentional change in runc:

@github-project-automation github-project-automation bot moved this from New to Done Issue in K3s Development Dec 11, 2024
@robjackstewart
Copy link
Author

Thanks @maggie44 , great find!

@ClashTheBunny
Copy link

Is it possible to revert this change in a point release (1.31.2 -> 1.31.3)? Going to a major API breaking change at a point release seems like a bad idea. There is also talk of reverting the change.

See my other comment about the long term security implications of this:
opencontainers/runc#3468 (comment)

@brandond
Copy link
Member

No. As I said on the other issue you commented on, we can update runc again if they decide to change behavior in a new release, but we are not planning to revert the version bump here.

@SuzukiHonoka
Copy link

No. As I said on the other issue you commented on, we can update runc again if they decide to change behavior in a new release, but we are not planning to revert the version bump here.

They have reverted the commit which casuing the issue. #4556
Many thanks for your attention, and we are looking forward to seeing you guys update the runc version once it is released.

@brandond
Copy link
Member

brandond commented Jan 2, 2025

We'll keep an eye out for v1.2.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done Issue
Development

No branches or pull requests

5 participants