Rootless Docker: Permission denied when creating work tree dir #124

schrammae · 2024-07-03T15:34:04Z

Hi.

For security reasons, we're using Docker in rootless mode. As opposed to default Docker mode, kas fails to do builds due to permission problems when creating the work tree dirs.

$ lsb_release -d
No LSB modules are available.
Description:    Debian GNU/Linux 12 (bookworm)

$ docker --version
Docker version 27.0.3, build 7d4bcd8

$ docker context show
rootless

$ systemctl --user status docker.service
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/usr/local/lib/systemd/user/docker.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-07-01 12:00:18 CEST; 2 days ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 1287 (rootlesskit)
      Tasks: 64
     Memory: 1.3G
        CPU: 2min 4.461s
     CGroup: /user.slice/user-2117.slice/[email protected]/app.slice/docker.service
             ├─1287 rootlesskit --state-dir=/run/user/2117/dockerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --po>
             ├─1325 /proc/self/exe --state-dir=/run/user/2117/dockerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback ->
             ├─1474 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 1325 tap0
             ├─1521 dockerd --config-file=/etc/docker/daemon-rootless.json
             └─1836 containerd --config /run/user/2117/docker/containerd/containerd.toml --log-level warn

kas ±|master|$ KAS_VERBOSE=1 kas-container --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas/
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
2024-07-03 14:27:07 - INFO     - kas 4.4 started
2024-07-03 14:27:07 - DEBUG    - Using selector: EpollSelector
2024-07-03 14:27:07 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-03 14:27:07 - DEBUG    - /repo/examples$ hg root
2024-07-03 14:27:07 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-03 14:27:07 - DEBUG    - /repo/examples$ hg root
2024-07-03 14:27:07 - DEBUG    - execute setup_dir
2024-07-03 14:27:07 - DEBUG    - execute setup_home
2024-07-03 14:27:07 - DEBUG    - execute init_setup_repos
2024-07-03 14:27:07 - DEBUG    - execute repo_setup_loop
2024-07-03 14:27:07 - DEBUG    - Loop repo_setup_loop: execute setup_repos_step
2024-07-03 14:27:07 - DEBUG    - execute finish_setup_repos
2024-07-03 14:27:07 - INFO     - Cloning repository poky
2024-07-03 14:27:07 - DEBUG    - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-03 14:27:07 - ERROR    - Command "/work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky" failed
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied

2024-07-03 14:27:07 - ERROR    - fetch repos failed: error code 128

Running it manually:

kas ±|master|$ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 /bin/bash

builder@6b8b35f6ab26:/repo$  id
uid=2117(builder) gid=100(users) groups=100(users)

builder@6b8b35f6ab26:/repo$ ls -lad /work/
drwxr-xr-x 12 root root 4096 Jul  3 14:26 /work/

builder@6b8b35f6ab26:/repo$ mkdir /work/poky
mkdir: cannot create directory ‘/work/poky’: Permission denied

Just for testing, I tried running with --docker-args "-e USER_ID=0 -e GROUP_ID=0":

kas ±|master|$ KAS_VERBOSE=1 kas-container --docker-args "-e USER_ID=0 -e GROUP_ID=0" --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas/
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none --user=root -e USER_ID=0 -e GROUP_ID=0 ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
2024-07-03 14:46:48 - INFO     - kas 4.4 started
2024-07-03 14:46:48 - DEBUG    - Using selector: EpollSelector
2024-07-03 14:46:48 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-03 14:46:48 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-03 14:46:48 - DEBUG    - execute setup_dir
2024-07-03 14:46:48 - DEBUG    - execute setup_home
2024-07-03 14:46:48 - DEBUG    - execute init_setup_repos
2024-07-03 14:46:48 - DEBUG    - execute repo_setup_loop
2024-07-03 14:46:48 - DEBUG    - Loop repo_setup_loop: execute setup_repos_step
2024-07-03 14:46:48 - DEBUG    - execute finish_setup_repos
2024-07-03 14:46:48 - INFO     - Cloning repository poky
2024-07-03 14:46:48 - DEBUG    - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-03 14:49:51 - DEBUG    - /work/poky$ git remote set-url origin https://git.yoctoproject.org/poky.git
2024-07-03 14:49:51 - DEBUG    - /work/poky$ git cat-file -t fb91a49387cfb0c8d48303bb3354325ba2a05587
2024-07-03 14:49:51 - INFO     - Repository poky already contains fb91a49387cfb0c8d48303bb3354325ba2a05587 as commit
2024-07-03 14:49:51 - DEBUG    - Configuration from config file:
{'distro': 'poky',
 'header': {'version': 14},
 'machine': 'qemux86-64',
 'repos': {'poky': {'commit': 'fb91a49387cfb0c8d48303bb3354325ba2a05587',
                    'layers': {'meta': None, 'meta-poky': None},
                    'url': 'https://git.yoctoproject.org/poky.git'}},
 'target': 'zlib-native'}
2024-07-03 14:49:51 - DEBUG    - execute repos_checkout
2024-07-03 14:49:51 - DEBUG    - /work/poky$ git status -s
2024-07-03 14:49:51 - DEBUG    - /work/poky$ git checkout -q fb91a49387cfb0c8d48303bb3354325ba2a05587
2024-07-03 14:49:51 - INFO     - Repository poky checked out to fb91a49387cfb0c8d48303bb3354325ba2a05587
2024-07-03 14:49:51 - DEBUG    - execute repos_apply_patches
2024-07-03 14:49:51 - DEBUG    - execute setup_environ
2024-07-03 14:49:51 - DEBUG    - /work/poky$ /tmp/tmpr2a3k6ky/get_bb_env /build
2024-07-03 14:49:51 - DEBUG    - execute write_bbconfig
2024-07-03 14:49:51 - DEBUG    - execute build
2024-07-03 14:49:51 - INFO     - /build$ /work/poky/bitbake/bin/bitbake -c build zlib-native
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Do not use Bitbake as root.

Summary: There was 1 ERROR message, returning a non-zero exit code.
2024-07-03 14:49:53 - ERROR    - Command "/work/poky/bitbake/bin/bitbake -c build zlib-native" failed with error 1

Testing further and silencing this error via touch build/conf/sanity.conf works to build the example. However, it fails on more complex builds with fatal: detected dubious ownership in repository:

kas ±|master ✗|$ KAS_VERBOSE=1 kas-container --docker-args "-e USER_ID=0 -e GROUP_ID=0" --log-level debug shell -c 'bitbake binutils' examples/openembedded.yml
...
ERROR: binutils-cross-x86_64-2.42-r0 do_unpack: Bitbake Fetcher Error: FetchError('Fetch command export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/work/poky/scripts/cross-intercept:/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/work/poky/scripts:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/usr/bin/x86_64-poky-linux/crossscripts:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/usr/sbin:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/usr/bin:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/sbin:/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/recipe-sysroot-native/bin:/work/poky/bitbake/bin:/build/tmp/hosttools"; export HOME="/tmp/tmpis75s0_d"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all remote set-url origin https://sourceware.org/git/binutils-gdb.git failed with exit code 128, output:\nfatal: detected dubious ownership in repository at \'/build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/git\'\nTo add an exception for this directory, call:\n\n\tgit config --global --add safe.directory /build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/git\n', None)
ERROR: Logfile of failure stored in: /build/tmp/work/x86_64-linux/binutils-cross-x86_64/2.42/temp/log.do_unpack.25770
ERROR: Task (/build/../work/poky/meta/recipes-devtools/binutils/binutils-cross_2.42.bb:do_unpack) failed with exit code '1'
ERROR: binutils-2.42-r0 do_unpack: Bitbake Fetcher Error: FetchError('Fetch command export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/work/poky/scripts:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot/usr/bin/crossscripts:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot-native/usr/sbin:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot-native/usr/bin:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot-native/sbin:/build/tmp/work/core2-64-poky-linux/binutils/2.42/recipe-sysroot-native/bin:/work/poky/bitbake/bin:/build/tmp/hosttools"; export HOME="/tmp/tmpis75s0_d"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all remote set-url origin https://sourceware.org/git/binutils-gdb.git failed with exit code 128, output:\nfatal: detected dubious ownership in repository at \'/build/tmp/work/core2-64-poky-linux/binutils/2.42/git\'\nTo add an exception for this directory, call:\n\n\tgit config --global --add safe.directory /build/tmp/work/core2-64-poky-linux/binutils/2.42/git\n', None)
ERROR: Logfile of failure stored in: /build/tmp/work/core2-64-poky-linux/binutils/2.42/temp/log.do_unpack.25771
ERROR: Task (/build/../work/poky/meta/recipes-devtools/binutils/binutils_2.42.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 500 tasks of which 22 didn't need to be rerun and 2 failed.

Summary: 2 tasks failed:
  /build/../work/poky/meta/recipes-devtools/binutils/binutils-cross_2.42.bb:do_unpack
  /build/../work/poky/meta/recipes-devtools/binutils/binutils_2.42.bb:do_unpack
Summary: There were 2 ERROR messages, returning a non-zero exit code.
2024-07-03 15:07:24 - ERROR    - Shell returned non-zero exit status
2024-07-03 15:07:24 - ERROR    - Command "/bin/bash -c 'bitbake binutils'" failed with error 1

So is it somehow possible to use kas with rootless Docker at all?

The text was updated successfully, but these errors were encountered:

fmoessbauer · 2024-07-03T15:43:41Z

Hi,

So is it somehow possible to use kas with rootless Docker at all?

we need to look into this, but unfortunately I don't have a rootless docker environment. Many of our users are using podman in rootless mode (including myself) and that definitely works. It can indeed be, that the user id mappings (which are only added for podman) are missing.

Please also have a look at the flags added to podman (just grep for podman in kas-container) and check if similar options exist for docker. Once we know what is missing, we can add this (given that there is a way to detect if docker is running in rootless mode).

schrammae · 2024-07-04T11:26:37Z

Hi @fmoessbauer,

thanks for your reply.

we need to look into this, but unfortunately I don't have a rootless docker environment. Many of our users are using podman in rootless mode (including myself) and that definitely works.

Can confirm - we used Podman before and are still partially using it. But Docker seems to offer better package support, i.e. one gets the latest Docker versions properly packaged for the current stable Debian/Ubuntu version. That's why our admins decided to go back to Docker and introduce rootless mode.

And they also don't want to maintain multiple container runtimes which is understandable. So at least for productive systems - which also covers the server I'd like to use for Yocto building - this means rootless Docker only.

Please also have a look at the flags added to podman (just grep for podman in kas-container) and check if similar options exist for docker.

I did, but besides --userns=keep-id I don't see anything relevant. --security-opt label=disable (1688d60):

This is for SELinux enabled systems only.

Once we know what is missing, we can add this (given that there is a way to detect if docker is running in rootless mode).

Detecting this is very easy:

the first Docker is my local one running as root
the second one is on the server running rootless Docker

$ docker context show
default

$ docker context show
rootless

It can indeed be, that the user id mappings (which are only added for podman) are missing.

I already stumbled upon that commit. The problem ist that --userns=keep-id is exclusive to Podman.

Disable namespace remapping for a container (--userns)

host is the only valid value for the --userns flag.

Digging around:

$ docker context inspect
[
    {
        "Name": "rootless",
        "Metadata": {
            "Description": "Rootless mode"
        },
        "Endpoints": {
            "docker": {
                "Host": "unix:///run/user/2117/docker.sock",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "/home/schramm/.docker/contexts/meta/12b961af5feb3e9d39f93b2cefb9a1a944f18d02cca0cac2f04f5a982240605f",
            "TLSPath": "/home/schramm/.docker/contexts/tls/12b961af5feb3e9d39f93b2cefb9a1a944f18d02cca0cac2f04f5a982240605f"
        }
    }
]

$ docker inspect a9eec6ca8825 | grep -E 'Privileged|UsernsMode|User'
            "Privileged": false,
            "UsernsMode": "",
            "User": "root",

Removing build/ and poky/ to clean the work dir, running with --userns=host:

--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied

Verifying the settings:

$ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none --user=root --userns=host ghcr.io/siemens/kas/kas:4.4 /bin/bas

builder@86e8baf1a2cd:/repo$ id
uid=2117(builder) gid=100(users) groups=100(users)

builder@86e8baf1a2cd:/repo$ ls -lad /work/
drwxr-xr-x 3 root root 4096 Jul  3 15:24 /work/

$ docker inspect 86e8baf1a2cd | grep -E 'Privileged|UsernsMode|User'
            "Privileged": false,
            "UsernsMode": "host",
            "User": "root",

Some research revealed Docker rootless mode without userns-remap.

Unfortunately, Docker in rootless mode does currently not support disabling the use of a user namespace; the option --userns=host is ignored.

Out of curiosity, testing without --user=root:

$ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none ghcr.io/siemens/kas/kas:4.4 /bin/bash
groupmod: Permission denied.
groupmod: cannot lock /etc/group; try again later.
usermod: user builder is currently used by process 1
chown: changing ownership of '/builder/.profile': Operation not permitted
chown: changing ownership of '/builder/.bash_logout': Operation not permitted
chown: changing ownership of '/builder/.bashrc': Operation not permitted
chown: changing ownership of '/builder': Operation not permitted
error: failed switching to "builder": operation not permitted

And running as root with -e USER_ID=0 -e GROUP_ID=0 is also not an option as this violates Bitbake's principles and makes Git barf about dubious file ownerships as shown above.

I'm not an expert in namespace mapping, but to me it looks like using kas with rootless Docker is currently not possible?

So this seems to sum it up perfectly:

Hope and pray that Docker will implement --userns=keep-id same as in Podman.

Silvanoc · 2024-07-08T07:11:54Z

Hi @schrammae,

If I remember right, the time podman was discussing rootless, they introduced fuse-overlayfs to overcome those issues.

Do you have it installed? It is anyway Debian recommended setup for rootless containers: https://docs.docker.com/engine/security/rootless/

schrammae · 2024-07-09T09:50:54Z

Hi @Silvanoc,

Thanks for your reply.

Do you have it installed? It is anyway Debian recommended setup for rootless containers: https://docs.docker.com/engine/security/rootless/

fuse-overlayfs is currently not installed on the server:

$ apt -qq list fuse-overlayfs
fuse-overlayfs/stable 1.10-1 amd64

The Docker config looks like this:

$ grep storage-driver /etc/docker/daemon-rootless.json
    "storage-driver": "overlay2"

$ docker info
...
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true

Checking the storage driver on a running container:

$ docker inspect --format='{{.Driver}}' e71a71cb6a8d
overlay2

Checking the docs: Docker storage drivers

fuse-overlayfs is preferred only for running Rootless Docker on a host that does not provide support for rootless overlay2. On Ubuntu and Debian 10, the fuse-overlayfs driver does not need to be used, and overlay2 works even in rootless mode.

So overlay2 seems to be ok and should not require fuse-overlayfs as far as I understand.

Known limitations

overlay2 (only if running with kernel 5.11 or later, or Ubuntu-flavored kernel)

The server is running Debian 6.1.94-1 (2024-06-21), so this should also be fine.

As I can't do any experiments on the server, I installed rootless Docker on my local machine. This is using Ubuntu 22.04.4 LTS and the Ubuntu guide does not mention fuse-overlayfs at all.

$ lsb_release -d
Description:    Ubuntu 22.04.4 LTS

$ docker context show
rootless

$ docker info
...
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true

This gives the same error as on Debian:

kas ±|master|$ KAS_VERBOSE=1 kas-container --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
2024-07-09 08:56:42 - INFO     - kas 4.4 started
2024-07-09 08:56:42 - DEBUG    - Using selector: EpollSelector
2024-07-09 08:56:42 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 08:56:42 - DEBUG    - /repo/examples$ hg root
2024-07-09 08:56:42 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 08:56:42 - DEBUG    - /repo/examples$ hg root
2024-07-09 08:56:42 - DEBUG    - execute setup_dir
2024-07-09 08:56:42 - DEBUG    - execute setup_home
2024-07-09 08:56:42 - DEBUG    - execute init_setup_repos
2024-07-09 08:56:42 - DEBUG    - execute repo_setup_loop
2024-07-09 08:56:42 - DEBUG    - Loop repo_setup_loop: execute setup_repos_step
2024-07-09 08:56:42 - DEBUG    - execute finish_setup_repos
2024-07-09 08:56:42 - INFO     - Cloning repository poky
2024-07-09 08:56:42 - DEBUG    - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-09 08:56:42 - ERROR    - Command "/work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky" failed
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied

2024-07-09 08:56:42 - ERROR    - fetch repos failed: error code 128

Creating a configuration file and testing with fuse-overlayfs

$ docker info
...
Storage Driver: fuse-overlayfs

$ apt -qq list fuse-overlayfs
fuse-overlayfs/jammy,now 1.7.1-1 amd64 [installed]

But this results in the same error as with overlay2:

kas ±|master|$ KAS_VERBOSE=1 kas-container --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
Unable to find image 'ghcr.io/siemens/kas/kas:4.4' locally
4.4: Pulling from siemens/kas/kas
09f376ebb190: Pull complete
2fc2fe69fbd5: Pull complete
3a271d2e7af8: Pull complete
701941e108db: Pull complete
b775f8c62321: Pull complete
132ee506e289: Pull complete
5e4f5ae397c3: Pull complete
54c27b899183: Pull complete
Digest: sha256:0bd8643166f22ad154d8a1cf9e56e9fb94fc08f6e03f42bba0110da8860b554f
Status: Downloaded newer image for ghcr.io/siemens/kas/kas:4.4
2024-07-09 09:05:52 - INFO     - kas 4.4 started
2024-07-09 09:05:52 - DEBUG    - Using selector: EpollSelector
2024-07-09 09:05:52 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 09:05:52 - DEBUG    - /repo/examples$ hg root
2024-07-09 09:05:52 - DEBUG    - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 09:05:52 - DEBUG    - /repo/examples$ hg root
2024-07-09 09:05:52 - DEBUG    - execute setup_dir
2024-07-09 09:05:52 - DEBUG    - execute setup_home
2024-07-09 09:05:52 - DEBUG    - execute init_setup_repos
2024-07-09 09:05:52 - DEBUG    - execute repo_setup_loop
2024-07-09 09:05:52 - DEBUG    - Loop repo_setup_loop: execute setup_repos_step
2024-07-09 09:05:52 - DEBUG    - execute finish_setup_repos
2024-07-09 09:05:52 - INFO     - Cloning repository poky
2024-07-09 09:05:52 - DEBUG    - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-09 09:05:52 - ERROR    - Command "/work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky" failed
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied

2024-07-09 09:05:52 - ERROR    - fetch repos failed: error code 128

Running it manually:

$ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 /bin/bash

$ docker inspect --format='{{.Driver}}' e514522b20f6
fuse-overlayfs

builder@e514522b20f6:/repo$ id
uid=2117(builder) gid=100(users) groups=100(users)

builder@e514522b20f6:/repo$ ls -lad /work/
drwxr-xr-x 12 root root 4096 Jul  2 12:45 /work/

builder@e514522b20f6:/repo$ mkdir /work/poky
mkdir: cannot create directory ‘/work/poky’: Permission denied

fmoessbauer · 2025-01-12T12:07:43Z

Hi @schrammae ,

I finally found some time to further analyze this and provide a workaround for docker rootless. I thought, I could implement this similar to our podman support, but unfortunately docker rootless does not support to share the user namespace with the host. By that all bind mounts belong to root inside the container, which is != the builder user. This is a bummer.

Anyways, I was able to come up with a workaround that requires a distinct KAS_WORK_DIR which is only written to by the container. For details, see https://groups.google.com/g/kas-devel/c/3yOzh1Br5jI

As this also requires changes to the container entrypoint, you need to build the container by yourself, or just wait until @jan-kiszka picks it up on next.

Docker rootless mode is similar to podman rootless mode, except that it does not support to share the userid namespace. By that, the bind mounted directories (like /repo, /work, /build) which are owned by the calling user, are mapped with uid==gid==0 inside the container. While we could align this by running as root inside the container, this is not an option as bitbake does not allow this. This comes with the following limitations: - /repo must be mounted ro to not destroy the uid mappings on the host - /work, /build must be an exclusive dir on the host that is only written to by the tooling inside kas-container - a git safe.dirs exception is needed as git operates as builder on repos owned by root - only operations that do not strictly require /repo:rw are supported. - ISAR mode is not supported in rootless mode, fallback to system docker Closes: siemens#124 Signed-off-by: Felix Moessbauer <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootless Docker: Permission denied when creating work tree dir #124

Rootless Docker: Permission denied when creating work tree dir #124

schrammae commented Jul 3, 2024

fmoessbauer commented Jul 3, 2024

schrammae commented Jul 4, 2024

Silvanoc commented Jul 8, 2024

schrammae commented Jul 9, 2024

fmoessbauer commented Jan 12, 2025

Rootless Docker: Permission denied when creating work tree dir #124

Rootless Docker: Permission denied when creating work tree dir #124

Comments

schrammae commented Jul 3, 2024

fmoessbauer commented Jul 3, 2024

schrammae commented Jul 4, 2024

Silvanoc commented Jul 8, 2024

schrammae commented Jul 9, 2024

fmoessbauer commented Jan 12, 2025