Security xattrs cause copy failures in builds #4675
Replies: 24 comments
-
|
@giuseppe PTAL |
Beta Was this translation helpful? Give feedback.
-
|
I am not able to reproduce on Fedora. Could you please provide a reproducer? I've created the tarball like: And Could you also please strace the buildah process (strace -o /tmp/buildah.log -f -s 1000 -v buildah bud ... |
Beta Was this translation helpful? Give feedback.
-
|
Sure thing! I gzipped it since it's otherwise 48M, and gzipped is 4M: My suspicion is it might be some feature that's disabled in It's also worth including the full output from In case that doesn't work, I figure I might as well provide some context to what I'm doing, since it might help debug the issue. Normally I would be able to just link the source for the project I'm working on, but since my gitea instance is offline, I'll make a GitHub repo for it: https://github.com/clarfonthey/cfg-system The entire source is a mess since I'm essentially rewriting something else I made, but the relevant thing I use to create the tarball is this: build/bootstrap.%.tar: src/exclude.% src/exclude.any src/pkgs/bootstrap.% src/pkgs/bootstrap.any cache/pkg build/%/etc/pacman.conf build/%/var/key/pacman/gpg-agent.conf build/%/var/key/pacman/gpg.conf | build/%/var/cache/pacman/pkg build/%/var/lib/pacman
unshare --map-root-user --map-auto \
pacman \
--root build/$*/ \
--config build/$*/etc/pacman.conf \
--dbpath build/$*/var/lib/pacman \
--cachedir cache/pkg/ \
--noscriptlet \
--noconfirm \
--needed \
--sync \
--refresh \
$(shell cat src/pkgs/bootstrap.any src/pkgs/bootstrap.$*)
unshare --map-root-user --map-auto \
pacman \
--root build/$*/ \
--config build/$*/etc/pacman.conf \
--dbpath build/$*/var/lib/pacman \
--query \
--check --check
unshare --map-root-user --map-auto \
tar \
--xattrs \
--acls \
--exclude-from=src/exclude.any \
--exclude-from=src/exclude.$* \
--directory build/$* \
--create . \
--file build/bootstrap.$*.tarModifying this to be less-specific to what I'm doing, perhaps this might work to build inside an mkdir --parents build cache
unshare --map-root-user --map-auto \
pacman \
--root build/x86_64/ \
--config build/x86_64/etc/pacman.conf \
--dbpath build/x86_64/var/lib/pacman \
--cachedir cache/pkg/ \
--noscriptlet \
--noconfirm \
--needed \
--sync \
--refresh \
$(cat src/pkgs/bootstrap.any src/pkgs/bootstrap.x86_64)
unshare --map-root-user --map-auto \
pacman \
--root build/x86_64/ \
--config build/x86_64/etc/pacman.conf \
--dbpath build/x86_64/var/lib/pacman \
--query \
--check --check
unshare --map-root-user --map-auto \
tar \
--xattrs \
--acls \
--exclude-from=src/exclude.any \
--exclude-from=src/exclude.x86_64 \
--directory build/x86_64 \
--create . \
--file build/bootstrap.x86_64.tarIt would require the Note that I'm using |
Beta Was this translation helpful? Give feedback.
-
|
from the output above it seemed you were running as root, since the Could you show the Another reason why it might fail is if you are using tmpfs as your backing store: Could you verify what is the file system below your graphroot ( |
Beta Was this translation helpful? Give feedback.
-
|
That's very strange -- I definitely wasn't running as root. Here's the exact command I ran: strace -o /tmp/buildah.log -f -s 1000 -v podman build --file src/Containerfile --build-arg ARCH=x86_64 .I'm running inside ZFS, so, it should be using that as my graph root. I even made a dataset specifically for it: $ cat /etc/containers/storage.conf | rg 'graphroot ='
graphroot = "/var/lib/containers/storage"
$ zfs list /var/lib/containers/storage
NAME USED AVAIL REFER MOUNTPOINT
zroot/data/ephemeral/containers/storage 172K 313G 172K /var/lib/containers/storage
$ cd /var/lib/containers/storage && touch foo && podman unshare attr -s security.capability -V "\1\0\0\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0" foo
Attribute "security.capability" set to a 37 byte value for foo:
\1\0\0\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0 |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure it is using that graphroot when you run as rootless. Can you check the output of |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I should clarify that I checked both $ podman info | rg graphRoot:
graphRoot: /var/lib/containers/ltdk/storage
$ buildah info | rg GraphRoot
"GraphRoot": "/var/lib/containers/ltdk/storage",All of them are ZFS datasets: $ zfs list | rg /var/lib/containers
$ zfs list | rg /var/lib/containers 21:35:49
zroot/data/ephemeral/containers 2.70G 308G 166K /var/lib/containers
zroot/data/ephemeral/containers/cache 98K 308G 98K /var/lib/containers/cache
zroot/data/ephemeral/containers/ltdk 2.70G 308G 132K /var/lib/containers/ltdk
zroot/data/ephemeral/containers/ltdk/cache 110K 308G 110K /var/lib/containers/ltdk/cache
zroot/data/ephemeral/containers/ltdk/storage 2.70G 308G 2.70G /var/lib/containers/ltdk/storage
zroot/data/ephemeral/containers/storage 172K 308G 172K /var/lib/containers/storageAnd just to rerun the experiment from before: $ cd /var/lib/containers/ltdk/storage && touch foo && podman unshare attr -s security.capability -V "\1\0\0\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0" foo
Attribute "security.capability" set to a 37 byte value for foo:
\1\0\0\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0So, you're right that the one I shared isn't the same folder, but ultimately it doesn't matter for this specific problem: all of them are running on ZFS. |
Beta Was this translation helpful? Give feedback.
-
|
would it be possible for you to share the tarball you've created? |
Beta Was this translation helpful? Give feedback.
-
|
I've been trying to avoid that since it's a filesystem image and close to 1G, but I can if you think that would help. That's also partially why I provided the source that I used to generate it if you're willing to try yourself. I was going to also try running on the mainline kernel instead of hardened to see if this is affecting the behaviour, but I was certain it wouldn't since I'm not aware of any additional settings that could cause this. |
Beta Was this translation helpful? Give feedback.
-
|
Okay. Two things:
My minimal reproduction is with this FROM scratch
WORKDIR /
ADD test.tar /And this tarball: I also updated the description to put this for the reproduction. Also because I figured it might be worth sharing, I ran |
Beta Was this translation helpful? Give feedback.
-
|
No issue on XFS. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for providing the tarball. I am also unable to reproduce the problem (tried both with and without fuse-overlayfs). Do you see any difference if you run I guess you are running on a new enough kernel that you can use native overlay instead of fuse-overlayfs. |
Beta Was this translation helpful? Give feedback.
-
|
I don't have any problem blowing away all my containers/images since this is just on my desktop I use for building. Will try some more tests to see if I can see a difference. Since I'm on Arch, I have the latest kernel + libraries, although linux-hardened is usually a patch version or two off the mainline kernel, which I run by default. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I don't have any special config in host:
arch: amd64
buildahVersion: 1.29.0
cgroupControllers:
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.6-1
path: /usr/bin/conmon
version: 'conmon version 2.1.6, commit: 158b5421dbac6bda96b1457955cf2e3c34af29bc'
cpuUtilization:
idlePercent: 88.24
systemPercent: 3.74
userPercent: 8.02
cpus: 12
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: waverider
idMappings:
gidmap:
- container_id: 0
host_id: 984
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 6.1.14-hardened1-1-hardened
linkmode: dynamic
logDriver: journald
memFree: 1451032576
memTotal: 33610547200
networkBackend: netavark
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.8.1-1
path: /usr/bin/crun
version: |-
crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 0
swapTotal: 0
uptime: 23h 52m 31.00s (Approximately 0.96 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /home/ltdk/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/ltdk/storage
graphRootAllocated: 341292613632
graphRootUsed: 524288
graphStatus:
Backing Filesystem: zfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /var/lib/containers/ltdk/storage/volumes
version:
APIVersion: 4.4.2
Built: 1677255177
BuiltTime: Fri Feb 24 11:12:57 2023
GitCommit: 74afe26887f814d1c39925a1624851ef3590e79c-dirty
GoVersion: go1.20.1
Os: linux
OsArch: linux/amd64
Version: 4.4.2And the full output from |
Beta Was this translation helpful? Give feedback.
-
|
The issue seems to be with ZFS not supporting setting Xattrs. |
Beta Was this translation helpful? Give feedback.
-
|
It's worth mentioning that the actual dataset does support xattrs, and I do have them enabled: $ zfs get xattr /var/lib/containers/ltdk/storage
NAME PROPERTY VALUE SOURCE
zroot/data/ephemeral/containers/ltdk/storage xattr sa inherited from zrootAdditionally, with native ZFS support via root podman, it also does work: $ sudo podman build . 16:41:54
STEP 1/3: FROM scratch
STEP 2/3: WORKDIR /
--> 035e1398ef2
STEP 3/3: ADD test.tar /
COMMIT
--> 4a42ede0439
4a42ede04399335658e3d6d1a0f6578a52814b376bf358f6e23ec068399332db |
Beta Was this translation helpful? Give feedback.
-
|
I still don't get how the command Could you run
|
Beta Was this translation helpful? Give feedback.
-
|
Okay, here's something super weird -- I tried building new containerfiles which run I initially thought this was just an issue with my system, but I updated and rebooted and got the same issue, so, I have no idea what's going on here. The processes remain around even if I SIGKILL them until I reboot. The two containerfiles: FROM archlinux
WORKDIR /
RUN touch /newgidmap
RUN attr -s security.capability -V "\01\00\00\02\40\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\0a\00\00\00" /newgidmapand FROM archlinux
WORKDIR /
RUN touch /newgidmap
RUN setcap cap_setgid=ep newgidmapI've attached the straces for both of these and the strace for the |
Beta Was this translation helpful? Give feedback.
-
|
could you try disabling seccomp ( |
Beta Was this translation helpful? Give feedback.
-
|
Doesn't seem to have a difference; invoking |
Beta Was this translation helpful? Give feedback.
-
|
if you don't strace it, does the process work fine and you don't hit the issue? |
Beta Was this translation helpful? Give feedback.
-
|
Ah, nope, same result as without the flag. This is the output for the attr version: And for the setcap version: |
Beta Was this translation helpful? Give feedback.
-
|
It's worth commenting that the However, when I originally built the tarball, which was generated using a raw I'm not sure how exactly the internals of podman differ from the standard unshare command, but I noticed this and figured it would be worth mentioning. |
Beta Was this translation helpful? Give feedback.
-
|
if you are seeing the same error with attr and setpcap then I don't think it is a podman/buildah error, it looks more like something FS-specific. I am converting to a discussion. |
Beta Was this translation helpful? Give feedback.
-
Description
When trying to
ADDa tarball which containssecurity.capabilityattributes on files (in my case:/usr/bin/newgidmapfrom a system image), theADDoperation will fail with anoperation not supportederror.There are a few closed issues about this, so, I decided to open a new one: #2127 #2168 #3294
Steps to reproduce the issue:
For this
Containerfile, runpodman build:With this for
test.tar:test.tar.zip (because GitHub is obnoxious, I had to put the tarball into a ZIP archive to upload it)
Describe the results you received:
Describe the results you expected:
Copy should succeed, or the error should be changed to indicate that this is not possible under rootless podman (I don't believe it is; this may be a bug).
Output of
rpm -q buildahorapt list buildah:Output of
buildah version:Output of
podman versionif reporting apodman buildissue:Output of
cat /etc/*release:Output of
uname -a:Output of
cat /etc/containers/storage.conf:Beta Was this translation helpful? Give feedback.
All reactions