Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unlinkat /workspace/.git: permission denied #92

Open
cmoulliard opened this issue Nov 19, 2024 · 17 comments
Open

unlinkat /workspace/.git: permission denied #92

cmoulliard opened this issue Nov 19, 2024 · 17 comments

Comments

@cmoulliard
Copy link
Member

Issue

Lifecycle can build a java quarkus project but fail due to permission issues

2024-11-19T10:29:39+01:00   Removing source code
2024-11-19T10:29:39+01:00 unable to invoke layer creator
2024-11-19T10:29:39+01:00 unable to remove /workspace/.git
2024-11-19T10:29:39+01:00 unlinkat /workspace/.git: permission denied
2024-11-19T10:29:39+01:00 Timer: Builder ran for 50.526944526s and ended at 2024-11-19T09:29:39Z
2024-11-19T10:29:39+01:00 ERROR: failed to build: exit status 1

How to reproduce

git clone https://github.com/redhat-buildpacks/java-buildpack-example
cd java-buildpack-example
// Start podman rootless
// Change within the code => ".withDockerSocket("/run/user/501/podman/podman.sock")" with your socket path
mvn package exec:java

@BarDweller

@BarDweller
Copy link
Collaborator

Please attach a link to a gist of a full log run for this error. Thanks.

@cmoulliard
Copy link
Member Author

Please attach a link to a gist of a full log run for this error. Thanks.

https://gist.github.com/cmoulliard/61f8145698b73339fe7dfddc82456874

@BarDweller

@BarDweller
Copy link
Collaborator

Tried to reproduce on both podman (linux) and docker (linux).

Both cases ran, but failed with roughly this error..

2024-11-19T14:38:00.269737673Z       [INFO] ------------------------------------------------------------------------
2024-11-19T14:38:00.269760484Z       [INFO] BUILD SUCCESS
2024-11-19T14:38:00.269764352Z       [INFO] ------------------------------------------------------------------------
2024-11-19T14:38:00.270674982Z       [INFO] Total time:  9.710 s
2024-11-19T14:38:00.270693578Z       [INFO] Finished at: 2024-11-19T14:38:00Z
2024-11-19T14:38:00.270698396Z       [INFO] ------------------------------------------------------------------------
2024-11-19T14:38:00.304834917Z       
2024-11-19T14:38:00.304898059Z unable to invoke layer creator
2024-11-19T14:38:00.304911175Z unable to contribute application layer
2024-11-19T14:38:00.304915947Z unable to resolve artifacts
2024-11-19T14:38:00.304919756Z unable to find any built artifacts for pattern(s):
2024-11-19T14:38:00.304923779Z target/quarkus-app/lib/
2024-11-19T14:38:00.304927656Z target/quarkus-app/*.jar
2024-11-19T14:38:00.304931568Z target/quarkus-app/app/
2024-11-19T14:38:00.304935244Z target/quarkus-app/quarkus/
2024-11-19T14:38:00.305653570Z Timer: Builder ran for 12.272230269s and ended at 2024-11-19T14:38:00Z
2024-11-19T14:38:00.305667530Z ERROR: failed to build: exit status 1

Which is odd, because that implied the maven build was successful, but did not produce a quarkus app in the expected packaging format (fast jar).

I tried just building locally, and confirmed this, and then realized your sample project pom.xml is incorrect, as it uses maven's compiler plugin, not quarkus's. So the example project isn't a real quarkus project, it merely has some quarkus code in it, and the pom.xml is hand crafted and incomplete.

At the very least you need to remove

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
                <configuration>
                    <!-- `-proc:only`: Explicitly enable annotation processing. -->
                    <compilerArgument>-proc:full</compilerArgument>
                </configuration>
            </plugin>

And replace it with

            <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <goals>
                            <goal>build</goal>
                            <goal>generate-code</goal>
                            <goal>generate-code-tests</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

And both docker and podman (on linux) create a working image.

I cannot recreate the error related to being unable to remove .git, If this still occurs it may be a podman behavior unique to macos

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 19, 2024

Which is odd, because that implied the maven build was successful, but did not produce a quarkus app in the expected packaging format (fast jar).

Be sure that your quarkus java project includes

            <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <goals>
                            <goal>build</goal>
                            <goal>generate-code</goal>
                            <goal>generate-code-tests</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Remark: My project is up to date: https://github.com/redhat-buildpacks/java-buildpack-example/blob/main/pom.xml#L86-L101

@BarDweller
Copy link
Collaborator

Something odd with github today?

I git cloned that project at 09:12 today, and again at 09:44 on a different host .. both resulted in a local directory with a head at commit 5d413ad. Github history says you made commit 338f510 '6 hours ago', which would be like 4 o'clock in the morning.. so I should have ended up with the version above..

@cmoulliard
Copy link
Member Author

Screenshot 2024-11-19 at 18 13 07

@BarDweller
Copy link
Collaborator

I suspect https://www.githubstatus.com/incidents/b85tj3x6n4vz may have delayed your commits from being visible when I did the git clone...

Regardless... this is feeling like a podman permissions issue on macos.

Debugging it may be challenging, as the app volume is always deleted after each build.. to find out why it can't delete it, we need to see the permissions on the workspace dir, and the dirs underneath it that are created in that volume during the build... getting that info is tough, the sanest way to do it is to add in a custom buildpack that dumps the filesystem permissions, and insert it into the order before the maven build.. doing that means building a custom builder etc, which isn't trivial.

Could you attempt to build the project using the same builder image, and the pack cli ? if the same error occurs, that would strongly suggest it to be a podman issue. Also, if the error does re-occur with pack cli, we can use it to inject a buildpack to dump the /workspace permissions/ownership.

Failing all that, we can customise the library code to dump the app volume ownership/permissions before each phase is executed, but that's also fairly messy.

Worth noting that the app dir isn't just mounted into the container, it's copied into a volume, and the volume is used for the build (otherwise the delete source step would erase your local source code). During that copy step, the ownership should be updated from your local user id, to the build uid/gid required by the image. So when looking at ownership & permissions during debugging you should expect to see it under that ownership, not yours.

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 20, 2024

We can kill the process and access during a few moments the terminal of the container created using the following setup

{
  "Id": "780772e5f9a4d60619904e6afa8d3b3cf56cb5490e40ac31c00a110b2ba29f9b",
  "Created": "2024-11-20T07:15:25.101481812Z",
  "Path": "/cnb/lifecycle/extender",
  "Args": [
    "-uid",
    "1002",
    "-gid",
    "1000",
    "-app",
    "/workspace",
    "-layers",
    "/layers",
    "-platform",
    "/platform",
    "-log-level",
    "debug"
  ],
  "State": {
    "Status": "running",
    "Running": true,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 270064,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "2024-11-20T07:15:25.171382385Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
  },
  "Image": "sha256:fa2f6672787bd4ae3e61fa96ba21ce7e67bd654721c35a26299c36499c99fae7",
  "ResolvConfPath": "/run/user/501/containers/overlay-containers/780772e5f9a4d60619904e6afa8d3b3cf56cb5490e40ac31c00a110b2ba29f9b/userdata/resolv.conf",
  "HostnamePath": "/run/user/501/containers/overlay-containers/780772e5f9a4d60619904e6afa8d3b3cf56cb5490e40ac31c00a110b2ba29f9b/userdata/hostname",
  "HostsPath": "/run/user/501/containers/overlay-containers/780772e5f9a4d60619904e6afa8d3b3cf56cb5490e40ac31c00a110b2ba29f9b/userdata/hosts",
  "LogPath": "",
  "Name": "/sweet_meninsky",
  "RestartCount": 0,
  "Driver": "overlay",
  "Platform": "linux",
  "MountLabel": "system_u:object_r:container_file_t:s0:c1022,c1023",
  "ProcessLabel": "",
  "AppArmorProfile": "",
  "ExecIDs": [],
  "HostConfig": {
    "Binds": [
      "/run/user/501/podman/podman.sock:/var/run/docker.sock:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-app-rrvlyvwbak:/workspace:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-build-pttxyrocvl:/cache-dir:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-kaniko-krobmeypez:/kaniko:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-launch-xxnljihuwp:/launch-cache-dir:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-output-zgkngnuijy:/layers:rw,rprivate,nosuid,nodev,rbind",
      "buildpack-platform-xrnsefddqb:/platform:rw,rprivate,nosuid,nodev,rbind"
    ],
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "journald",
      "Config": null
    },
    "NetworkMode": "bridge",
    "PortBindings": {},
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "ConsoleSize": [
      0,
      0
    ],
    "Annotations": {
      "io.container.manager": "libpod",
      "io.podman.annotations.label": "disable",
      "org.opencontainers.image.stopSignal": "15",
      "org.systemd.property.KillSignal": "15",
      "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
    },
    "CapAdd": [],
    "CapDrop": [],
    "CgroupnsMode": "",
    "Dns": [],
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": [],
    "GroupAdd": [],
    "IpcMode": "shareable",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 200,
    "PidMode": "private",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": [
      "label=disable"
    ],
    "UTSMode": "private",
    "UsernsMode": "",
    "ShmSize": 65536000,
    "Runtime": "oci",
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "user.slice",
    "BlkioWeight": 0,
    "BlkioWeightDevice": null,
    "BlkioDeviceReadBps": null,
    "BlkioDeviceWriteBps": null,
    "BlkioDeviceReadIOps": null,
    "BlkioDeviceWriteIOps": null,
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": [],
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": 0,
    "OomKillDisable": false,
    "PidsLimit": 0,
    "Ulimits": [
      {
        "Name": "RLIMIT_NOFILE",
        "Hard": 524288,
        "Soft": 524288
      },
      {
        "Name": "RLIMIT_NPROC",
        "Hard": 51268,
        "Soft": 51268
      }
    ],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": null,
    "ReadonlyPaths": null
  },
  "GraphDriver": {
    "Data": {
      "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/c367706b28bae60bbcb9c3f8721b703f6248932a5043767b8cf6ff99c0a4bd0b/diff:/var/home/core/.local/share/containers/storage/overlay/1023c6a31a02c29fb9b0b07c5cdbf63cb0202c8e81030f5584a12d0e5e82e47b/diff:/var/home/core/.local/share/containers/storage/overlay/8a9ceadf9dcd36cb4b5b40eb147a12e5a1cb31c35aabc18077b68cadc762154a/diff:/var/home/core/.local/share/containers/storage/overlay/16d20db15ed5e4348867abe3db6be1aae1b1fd2dbab2e4b659259d9588fca6d8/diff:/var/home/core/.local/share/containers/storage/overlay/00e8b3cf04ce28b9271fc8f37b669a94a2bfde42c7786aec24a4967ea5b72b11/diff:/var/home/core/.local/share/containers/storage/overlay/fc957f3ba1188367353503b9d1887b83f7e414f66a3f3352f3ae2fdfb5f5d340/diff:/var/home/core/.local/share/containers/storage/overlay/e340a4f2094c754ac228dd65ba644e12596f8310be3ef4ba69aa58e9fe50cd09/diff:/var/home/core/.local/share/containers/storage/overlay/e94ff26cfd780fef778b5961c9d4bf1aea9f4ef328598ab7a3fe5bfce1e85ec2/diff:/var/home/core/.local/share/containers/storage/overlay/7e5ad9f36e8228498a1e786b504dc26c537a856cdeb4942f12d1ccfde344f825/diff:/var/home/core/.local/share/containers/storage/overlay/751cc3442a26f01f912ecd934d4a187075d77f1931a5b10de1f0534d59122880/diff:/var/home/core/.local/share/containers/storage/overlay/8765f74831bfffb5faf75ac4630d2cc60db8d017ceb70cfc0394b0201f55f531/diff:/var/home/core/.local/share/containers/storage/overlay/30de54452197469f2d530544f3d20e4c36a19ff44be6e9eaad65dfafc983c2aa/diff:/var/home/core/.local/share/containers/storage/overlay/91f9c4f4b733586629d8ea78fdd594a631c2686b7a0b6f4f3d21712dd498dac8/diff:/var/home/core/.local/share/containers/storage/overlay/938249427acedcbc3e8968771b2b61a8e061d6bf8539f4928d81c5d2ef3d6d23/diff:/var/home/core/.local/share/containers/storage/overlay/d13462a91fe76c2daf9d5e61acc6f244f002f8d163209639103565d5ddeb3681/diff:/var/home/core/.local/share/containers/storage/overlay/2620d3e9e06976b0bb00bf62c966cd8c94652a7af1d9351727695b88611aef02/diff:/var/home/core/.local/share/containers/storage/overlay/03b74ab548dddfa8c92f26b37c9f8cef6d12da9340bc15846de501226ea45f33/diff:/var/home/core/.local/share/containers/storage/overlay/2060084ce1dbfdb3128fce3a4a51ee7bc9377e80d5f0d9d208e7f035deeffb74/diff:/var/home/core/.local/share/containers/storage/overlay/77f2bbbea0e2dfcb3eb04fa468808efc4780769d03c2a19e676d77f54936da61/diff:/var/home/core/.local/share/containers/storage/overlay/5455171fc7e241c8ec10b88ae5a860fe6564555a452d72677a7bf313a0bc47b3/diff:/var/home/core/.local/share/containers/storage/overlay/40abe965d2815d6f93e4ce8c1b28af9a0b377d7dcd7f49d3adefc1e21ef92658/diff:/var/home/core/.local/share/containers/storage/overlay/4c7da61b049d4ab64cc225d1258b9b85af5758a7a0a49a148d688e8515e8b6ec/diff:/var/home/core/.local/share/containers/storage/overlay/ac6c5097ff146cdecf5e324f409193c89209a31f5ac4c3074a3086dd75473905/diff:/var/home/core/.local/share/containers/storage/overlay/b1df75aa60c6ed3a534d78e66084321c18eaa0f64e0635d0ea5fcf180ad906fe/diff:/var/home/core/.local/share/containers/storage/overlay/c29482164c940ddbee1dfc97716940f5fb64ed9721029b09023ba00939d066fc/diff:/var/home/core/.local/share/containers/storage/overlay/ba5acfb460b43c59ef21c7460fb20414258c9e0b2cdc0a08f5b0d8c1622d58b8/diff:/var/home/core/.local/share/containers/storage/overlay/0df988b2fa733833a87a76d42aabdf14fd6147543566441a0f4954bb5f691516/diff:/var/home/core/.local/share/containers/storage/overlay/23dbba15fa752289083be211f861461cd8a6e5084af443d5d12c1fec34f38a7f/diff:/var/home/core/.local/share/containers/storage/overlay/79e27353e1d2e12a2cb26f9edebebdbbad4ea4973fc67e5276b0a60dd440e065/diff:/var/home/core/.local/share/containers/storage/overlay/d273278511a6f33f26de929f370cb5d48e20752f5b6ada33d3f65baeff270d2a/diff:/var/home/core/.local/share/containers/storage/overlay/193472acf7857c0fcd7a9f637103d4953d4cd89c4dc8d0cbf9c1aedd15571db2/diff:/var/home/core/.local/share/containers/storage/overlay/9d325423723b5411317c7192500bbc608da5632ad34c25d63359e4409195784d/diff:/var/home/core/.local/share/containers/storage/overlay/1b3ff95b49461864a7069915c833edc1465ae77aa3aa652cee97a157a031235e/diff:/var/home/core/.local/share/containers/storage/overlay/d51cf1f24ec61a8055dda179c65f8513023d20eba557fa2cdde0750a42070eec/diff:/var/home/core/.local/share/containers/storage/overlay/b025a5e3422389819d0c316efd89bf2c50d8450dfab61817ffaa2f6a4c7eab5c/diff:/var/home/core/.local/share/containers/storage/overlay/2b8af842047e80a8623e1066baa2aa1a8a200fd4bc5dc5af9932bac79322ff21/diff:/var/home/core/.local/share/containers/storage/overlay/b3b0368643af5441e46c5d2fcdf7e38aff226ca3b71097618525f1a81ad535c6/diff:/var/home/core/.local/share/containers/storage/overlay/321ee30876c372a891add3bb18f59c8da28ba44dfbbc708b416f272903744b07/diff:/var/home/core/.local/share/containers/storage/overlay/3faccbb478905c2c991d51a74a0013c5055e8f080d4290d85d4087b843b727ea/diff:/var/home/core/.local/share/containers/storage/overlay/ea21388961c54aa13a9a7766b67fa80f057df174f7b6027f72ee76fb19751c39/diff:/var/home/core/.local/share/containers/storage/overlay/4f845b3bafc390da466024d9eb3d69f51ac12c475c70e41ca673c559d1a3c572/diff:/var/home/core/.local/share/containers/storage/overlay/433b8805eb3e886d61e76a954a2b50da00d850162d6b00c57a49aab762d3b18d/diff:/var/home/core/.local/share/containers/storage/overlay/3260b35b020b8ac934d70702ff0db680d9576daf04e2f8e14b82fffb9e2ae524/diff:/var/home/core/.local/share/containers/storage/overlay/65df6d7c54a7f1f2d8627d22d3377c81c3072e597501560c807aee0deca0ca08/diff:/var/home/core/.local/share/containers/storage/overlay/44ab5e3badbe4e9f5e6e117b1dbc4c2bc3575995f26814049da3afea906af04b/diff:/var/home/core/.local/share/containers/storage/overlay/6632c7d845d52f2488999d6063ee023f5ed7eba885e023814468e583acd55ea3/diff:/var/home/core/.local/share/containers/storage/overlay/3f40b6b9c0d11614bbdbc84567e6a517580c219d41a4251fd0817543252b0324/diff:/var/home/core/.local/share/containers/storage/overlay/231e4444d72e39d555581039a9983ae790f56e22b116c87a5f83ce02c68a19db/diff:/var/home/core/.local/share/containers/storage/overlay/c0adc838f16d125be0ef641134cb894dad2a4741f1e755f5cd3e01c8c07fe231/diff:/var/home/core/.local/share/containers/storage/overlay/cb973d48271cfb4bad03e3ef5f9e1513164b6aff04e4180207657d5aa2b3cd6b/diff",
      "MergedDir": "/var/home/core/.local/share/containers/storage/overlay/d4da7753ac6f91f01fadf8fb0538ef81caa0abec0671efc2f00d3cb756567d36/merged",
      "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/d4da7753ac6f91f01fadf8fb0538ef81caa0abec0671efc2f00d3cb756567d36/diff",
      "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/d4da7753ac6f91f01fadf8fb0538ef81caa0abec0671efc2f00d3cb756567d36/work"
    },
    "Name": "overlay"
  },
  "SizeRootFs": 0,
  "Mounts": [
    {
      "Type": "volume",
      "Name": "buildpack-app-rrvlyvwbak",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-app-rrvlyvwbak/_data",
      "Destination": "/workspace",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "buildpack-platform-xrnsefddqb",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-platform-xrnsefddqb/_data",
      "Destination": "/platform",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "buildpack-output-zgkngnuijy",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-output-zgkngnuijy/_data",
      "Destination": "/layers",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "buildpack-kaniko-krobmeypez",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-kaniko-krobmeypez/_data",
      "Destination": "/kaniko",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "buildpack-build-pttxyrocvl",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-build-pttxyrocvl/_data",
      "Destination": "/cache-dir",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "buildpack-launch-xxnljihuwp",
      "Source": "/var/home/core/.local/share/containers/storage/volumes/buildpack-launch-xxnljihuwp/_data",
      "Destination": "/launch-cache-dir",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/run/user/501/podman/podman.sock",
      "Destination": "/var/run/docker.sock",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    }
  ],
  "Config": {
    "Hostname": "780772e5f9a4",
    "Domainname": "",
    "User": "0",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "CNB_REGISTRY_AUTH={}",
      "CNB_EXPERIMENTAL_MODE=warn",
      "CNB_GROUP_ID=1000",
      "CNB_STACK_ID=io.buildpacks.stacks.ubi8",
      "CNB_USER_ID=1002",
      "CNB_PLATFORM_API=0.10",
      "container=oci",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "BP_JVM_VERSION=21",
      "CNB_LOG_LEVEL=trace",
      "HOME=/root",
      "HOSTNAME=780772e5f9a4"
    ],
    "Cmd": [
      "/cnb/lifecycle/extender",
      "-uid",
      "1002",
      "-gid",
      "1000",
      "-app",
      "/workspace",
      "-layers",
      "/layers",
      "-platform",
      "/platform",
      "-log-level",
      "debug"
    ],
    "Image": "fa2f6672787bd4ae3e61fa96ba21ce7e67bd654721c35a26299c36499c99fae7",
    "Volumes": null,
    "WorkingDir": "/layers",
    "Entrypoint": [
      ""
    ],
    "OnBuild": null,
    "Labels": {
      "architecture": "x86_64",
      "build-date": "2024-10-30T23:56:36",
      "com.redhat.component": "ubi8-minimal-container",
      "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
      "description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
      "distribution-scope": "public",
      "io.buildah.version": "1.33.8",
      "io.buildpacks.builder.metadata": "{\"description\":\"Ubi 8 base image with buildpacks for Node.js, Java, Quarkus and ...github.com/paketo-community/ubi-nodejs-extension\",\"name\":\"Ubi Node.js Extension\"}}}",
      "io.buildpacks.stack.description": "base build ubi8 image to support buildpacks",
      "io.buildpacks.stack.distro.name": "rhel",
      "io.buildpacks.stack.distro.version": "8.10",
      "io.buildpacks.stack.homepage": "https://github.com/paketo-community/ubi-base-stack",
      "io.buildpacks.stack.id": "io.buildpacks.stacks.ubi8",
      "io.buildpacks.stack.maintainer": "Paketo Community",
      "io.buildpacks.stack.metadata": "{}",
      "io.buildpacks.stack.mixins": "null",
      "io.buildpacks.stack.released": "2024-11-18T10:45:35Z",
      "io.k8s.description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
      "io.k8s.display-name": "Red Hat Universal Base Image 8 Minimal",
      "io.openshift.expose-services": "",
      "io.openshift.tags": "minimal rhel8",
      "maintainer": "Red Hat, Inc.",
      "name": "ubi8-minimal",
      "release": "1130",
      "summary": "Provides the latest release of the minimal Red Hat Universal Base Image 8.",
      "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.10-1130",
      "vcs-ref": "4f8da2b64a13f2a264bd802d8909bf803211fb20",
      "vcs-type": "git",
      "vendor": "Red Hat, Inc.",
      "version": "8.10"
    },
    "StopSignal": "15",
    "StopTimeout": 10
  },
  "NetworkSettings": {
    "Bridge": "",
    "SandboxID": "",
    "SandboxKey": "/run/user/501/netns/netns-1b016dc0-20e4-7a3a-2049-7a0eb9e61130",
    "Ports": {},
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "10.88.0.1",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "10.88.0.46",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "MacAddress": "ee:b3:00:0d:d7:a8",
    "Networks": {
      "podman": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": [
          "780772e5f9a4"
        ],
        "MacAddress": "ee:b3:00:0d:d7:a8",
        "DriverOpts": null,
        "NetworkID": "podman",
        "EndpointID": "",
        "Gateway": "10.88.0.1",
        "IPAddress": "10.88.0.46",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DNSNames": null
      }
    }
  }
}

The /workspace user and owner are cnb

[root@5bac8fed9093 layers]# ls -la ../workspace/
total 48
drwxr-xr-x.  5 root root   161 Nov 20 08:05 .
dr-xr-xr-x.  1 root root    73 Nov 20 08:06 ..
-rwxr-xr-x.  1 cnb  cnb     75 Nov 20 08:05 .dockerignore
-rwxr-xr-x.  1 cnb  cnb    396 Nov 20 08:05 .gitignore
drwxr-xr-x.  3 cnb  cnb     21 Nov 20 08:05 .mvn
-rwxr-xr-x.  1 cnb  cnb   2193 Nov 20 08:05 README.md
-rwxr-xr-x.  1 cnb  cnb    251 Nov 20 08:05 id-debug.sh
-rwxr-xr-x.  1 cnb  cnb  11289 Nov 20 08:05 mvnw
-rwxr-xr-x.  1 cnb  cnb   7591 Nov 20 08:05 mvnw.cmd
-rwxr-xr-x.  1 cnb  cnb   5017 Nov 20 08:05 pom.xml
drwxr-xr-x.  4 cnb  cnb     30 Nov 20 08:05 src
drwxr-xr-x. 10 cnb  cnb   4096 Nov 20 08:05 target

[root@5bac8fed9093 layers]# ls -la 
total 12
drwxr-xr-x. 3 cnb  cnb    79 Nov 20 08:05 .
dr-xr-xr-x. 1 root root   73 Nov 20 08:06 ..
-rw-r--r--. 1 cnb  cnb   561 Nov 20 08:06 analyzed.toml
drwxr-xr-x. 4 cnb  cnb    30 Nov 20 08:05 generated
-rw-r--r--. 1 cnb  cnb  1807 Nov 20 08:05 group.toml
-rw-r--r--. 1 cnb  cnb  2648 Nov 20 08:05 plan.toml

kaniko

[root@5bac8fed9093 layers]# ls -la ../kaniko/
total 0
drwxr-xr-x. 3 cnb  cnb   19 Nov 20 08:06 .
dr-xr-xr-x. 1 root root 155 Nov 20 08:06 ..
drwxr-xr-x. 4 cnb  cnb   32 Nov 20 08:06 cache
[root@5bac8fed9093 layers]# ls -la ../kaniko/cache/
total 4
drwxr-xr-x. 4 cnb  cnb    32 Nov 20 08:06 .
drwxr-xr-x. 3 cnb  cnb    19 Nov 20 08:06 ..
drwxr-xr-x. 3 cnb  cnb    85 Nov 20 08:06 base
drwxr-xr-x. 6 root root 4096 Nov 20 08:06 layers
[root@5bac8fed9093 layers]# ls -la ../kaniko/cache/layers/
total 4
drwxr-xr-x. 6 root root 4096 Nov 20 08:06 .
drwxr-xr-x. 4 cnb  cnb    32 Nov 20 08:06 ..
drwxr-xr-x. 3 root root   55 Nov 20 08:06 cached:18c5665423525b1a482370f84b32f5476a95e344ff1972bf3c4bda1c65f7bee9
drwxr-xr-x. 3 root root   55 Nov 20 08:06 cached:312d193799d170300bf75e7a4b6e530820647888f1f300fe9f554da5a9941e0f
drwxr-xr-x. 3 root root   55 Nov 20 08:06 cached:4b6d85618cc7760744c5951a670c198045c12e58a6274e4893aaa2203a65d6b0
drwxr-xr-x. 3 root root   55 Nov 20 08:06 cached:8710f127fae6189d5b39cba38404d2f3fee468d2da1ca5c97df6d0cbe4976f67

except for /cnb which is owned by root

[root@5bac8fed9093 layers]# ls -la ../cnb
total 16
drwxr-xr-x. 1 root root   22 Nov 19 12:59 .
dr-xr-xr-x. 1 root root  155 Nov 20 08:06 ..
drwxr-xr-x. 3 root root   17 Jan  1  1980 build-config
drwxr-xr-x. 1 root root   49 Nov 19 12:59 buildpacks
drwxr-xr-x. 1 root root   51 Nov 19 12:59 extensions
drwxr-xr-x. 2 root root 4096 Jan  1  1980 lifecycle
-rw-r--r--. 1 root root  673 Jan  1  1980 order.toml
-rw-r--r--. 1 root root   90 Jan  1  1980 run.toml
-rw-r--r--. 1 root root   91 Jan  1  1980 stack.toml

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 20, 2024

Podman - macos

We got the same error permission denied on the files located under /workspace no matter if I use on macos rootless or rootful

// Using rootful

❯ podman info | grep sock
    path: /run/podman/podman.sock

❯ mvn clean compile exec:java
~/code/redhat-buildpacks/buildpack-test on main •
2024-11-20T09:14:09+01:00   Removing source code
2024-11-20T09:14:09+01:00 unable to invoke layer creator
2024-11-20T09:14:09+01:00 unable to remove /workspace/.dockerignore
2024-11-20T09:14:09+01:00 unlinkat /workspace/.dockerignore: permission denied
2024-11-20T09:14:09+01:00 Timer: Builder ran for 52.119196464s and ended at 2024-11-20T08:14:09Z
2024-11-20T09:14:09+01:00 ERROR: failed to build: exit status 1

Docker - macos

That works on macos as files under /workspace have the proper user 1002 and group 1000 id

Screenshot 2024-11-20 at 10 26 45

@BarDweller
Copy link
Collaborator

The /workspace user and owner are cnb

[root@5bac8fed9093 layers]# ls -la ../workspace/
total 48
drwxr-xr-x.  5 root root   161 Nov 20 08:05 .
dr-xr-xr-x.  1 root root    73 Nov 20 08:06 ..
-rwxr-xr-x.  1 cnb  cnb     75 Nov 20 08:05 .dockerignore
...

No.. the owner of /workspace, is root/root (look at ownership for .), here's the problem, /workspace should be owned by cnb/cnb , since root owns /workspace, with perms 755, only root is allowed to create or delete files within the directory.. I've seen this before, but it's not supposed to happen since I created the ephemeral build image.

Curiously, the build still worked, because target was created, unless that already existed locally.. which I guess it would because you are driving the buildpack step after the regular build.. try just driving mvn clean exec:java and I suspect it will fail with some error about being able to create target/classes (or similar).

Lets confirm the ephemeral builder image created has correct ownership for /workspace, (there's no reason it shouldn't because the other dirs DO and they are created the same way)..

Run a podman build, kill the process before it tidies up and deletes the images/volumes, and launch a container based on the ephemeral builder image.. you will see that created in the log messages coming from the library (at the mo, you don't have any of those, because you don't have an slf4j log impl on your path, add org.slf4j:slf4j-simple:1.7.30 as a dep, and you should start seeing debug messages from the library code, including when it creates the ephemeral image, and what it's image id is (the image isn't named, it's referenced by id).. you'll see it refer to that builder image at each stage when it launches any phase container.

Verify the ownership for /workspace in the ephemeral build container.. if this is cnb/cnb as expected, then we have a podman bug (but we kinda knew that) .. and podman is not respecting the ownership of the mountpoint when mounting the app volume (odd, since kaniko's ownership IS being respected as cnb/cnb)

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 20, 2024

I added the missing slf4j-simple jar to the pom.xml and can grab more information which of course confirm that we know as maven Cannot create resource output directory: /workspace/target/classes -> [Help 1] due to permissions issue

2024-11-20T10:44:55+01:00       [INFO] BUILD FAILURE
2024-11-20T10:44:55+01:00       [INFO] ------------------------------------------------------------------------
2024-11-20T10:44:55+01:00       [INFO] Total time:  14.010 s
2024-11-20T10:44:55+01:00       [INFO] Finished at: 2024-11-20T09:44:55Z
2024-11-20T10:44:55+01:00       [INFO] ------------------------------------------------------------------------
2024-11-20T10:44:55+01:00       [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.3.1:resources (default-resources) on project quarkus-hello: Cannot create resource output directory: /workspace/target/classes -> [Help 1]
2024-11-20T10:44:55+01:00       [ERROR] 
2024-11-20T10:44:55+01:00       [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2024-11-20T10:44:55+01:00       [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2024-11-20T10:44:55+01:00       [ERROR] 
2024-11-20T10:44:55+01:00       [ERROR] For more information about the errors and possible solutions, please read the following articles:
2024-11-20T10:44:55+01:00       [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
2024-11-20T10:44:55+01:00 unable to invoke layer creator
2024-11-20T10:44:55+01:00 unable to contribute application layer
2024-11-20T10:44:55+01:00 error running build
2024-11-20T10:44:55+01:00 exit status 1
2024-11-20T10:44:55+01:00       Timer: Builder ran for 21.723414277s and ended at 2024-11-20T09:44:55Z
2024-11-20T10:44:55+01:00 ERROR: failed to build: exit status 1
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Extender - Extender container complete, with exit code 51
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - Post Build cleanup activites:
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - temporary build volumes removed

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 20, 2024

Lets confirm the ephemeral builder image created has correct ownership for /workspace, (there's no reason it shouldn't because the other dirs DO and they are created the same way)..

I pushed (using podman) on the start button of the ephemeral container created for cnb lifecycle extender and got the same cnb/cnb uid/gid

Screenshot 2024-11-20 at 10 48 52

If we launch a container using the pakero ubi-builder image, we observe

bash-4.4$ pwd
/layers
bash-4.4$ ls -la
total 0
drwxr-xr-x. 2 cnb  cnb   6 Jan  1  1980 .
dr-xr-xr-x. 1 root root 29 Nov 20 09:51 ..
bash-4.4$ ls -la ..
total 8
dr-xr-xr-x.   1 root   root     29 Nov 20 09:51 .
dr-xr-xr-x.   1 root   root     29 Nov 20 09:51 ..
lrwxrwxrwx.   1 root   root      7 Jun 21  2021 bin -> usr/bin
dr-xr-xr-x.   2 root   root      6 Jun 21  2021 boot
drwxr-xr-x.   1 root   root     22 Nov 19 12:59 cnb
drwxr-xr-x.   5 root   root    360 Nov 20 09:51 dev
drwxr-xr-x.   1 root   root     33 Nov 18 10:45 etc
drwxr-xr-x.   1 root   root     17 Nov 19 12:59 home
drwxr-xr-x.   2 cnb    cnb       6 Jan  1  1980 layers
lrwxrwxrwx.   1 root   root      7 Jun 21  2021 lib -> usr/lib
lrwxrwxrwx.   1 root   root      9 Jun 21  2021 lib64 -> usr/lib64
lrwxrwxrwx.   1 root   root     14 Jan  1  1980 lifecycle -> /cnb/lifecycle
drwx------.   2 root   root      6 Oct 30 23:59 lost+found
drwxr-xr-x.   2 root   root      6 Jun 21  2021 media
drwxr-xr-x.   2 root   root      6 Jun 21  2021 mnt
drwxr-xr-x.   2 root   root      6 Jun 21  2021 opt
drwxr-xr-x.   3 root   root     17 Jan  1  1980 platform
dr-xr-xr-x. 249 nobody nobody    0 Nov 20 09:51 proc
dr-xr-x---.   3 root   root   4096 Oct 31 00:03 root
drwxr-xr-x.   1 root   root     27 Nov 20 09:51 run
lrwxrwxrwx.   1 root   root      8 Jun 21  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root   root      6 Jun 21  2021 srv
dr-xr-xr-x.  12 nobody nobody    0 Nov 20 09:36 sys
drwxrwxrwt.   2 root   root     58 Oct 31 00:03 tmp
drwxr-xr-x.  12 root   root    144 Oct 30 23:59 usr
drwxr-xr-x.  19 root   root   4096 Oct 31 00:00 var
drwxr-xr-x.   2 cnb    cnb       6 Jan  1  1980 workspace

bash-4.4$ ls -la ../workspace/
total 0
drwxr-xr-x. 2 cnb  cnb   6 Jan  1  1980 .
dr-xr-xr-x. 1 root root 29 Nov 20 09:51 ..

bash-4.4$ ls -la ../cnb       
total 16
drwxr-xr-x. 1 root root   22 Nov 19 12:59 .
dr-xr-xr-x. 1 root root   29 Nov 20 09:51 ..
drwxr-xr-x. 3 root root   17 Jan  1  1980 build-config
drwxr-xr-x. 1 root root   49 Nov 19 12:59 buildpacks
drwxr-xr-x. 1 root root   51 Nov 19 12:59 extensions
drwxr-xr-x. 2 root root 4096 Jan  1  1980 lifecycle
-rw-r--r--. 1 root root  673 Jan  1  1980 order.toml
-rw-r--r--. 1 root root   90 Jan  1  1980 run.toml
-rw-r--r--. 1 root root   91 Jan  1  1980 stack.toml

@BarDweller
Copy link
Collaborator

The paketo ubi-builder image is irrelevant, we don't use that one to create the phase containers, we use the ephemeral builder image, the id of which is logged before each phase execution. We need to see the permissions from a fresh container created from that image.

The screenshot is relaunching one of the phase containers, and (I'm guessing) is also remounting the volumes, because /workspace has content under it, which is only ever present in the app volume.

Can you try to launch a container based on the ephemeral builder image id and verify the perms of the dirs in root with ls -al /

@cmoulliard
Copy link
Member Author

cmoulliard commented Nov 20, 2024

Can you try to launch a container based on the ephemeral builder image id and verify the perms of the dirs in root with ls -al /

No as the link to the ephemeral image don't exist anymore and/or I got such an error

// If I try to run the ephemeral image =>

❯ podman run -it fa2f6672787b
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
Error: preparing container cafa9f34ebb102668972893415f0c79de9d4ce158f01f75690e1aea59d6073d3 for attach: crun: open executable: Operation not permitted: OCI permission denied

@BarDweller
Copy link
Collaborator

You'll need to kill off the build once analyze phase is complete, by that point the ephemeral image is created, and you'll skip the tidy up part that deletes it normally

@cmoulliard
Copy link
Member Author

I was able to launch a container with the ephemeral image

❯ podman run -it 416d28a3d9116fb949081520fd8a1ee9c90ae9cc18b03a075265d584c1d7bdb0 bash
...
drwxr-xr-x.   1 cnb    cnb       6 Nov 20 10:34 workspace
[root@9313fa492b58 layers]# ls -la ../workspace/
total 0
drwxr-xr-x. 1 cnb  cnb   6 Nov 20 10:34 .
dr-xr-xr-x. 1 root root 29 Nov 20 10:39 ..

@cmoulliard
Copy link
Member Author

FYI. Here is the output of the commands executed using the ephemeral builder image with root or ui/gid = 1500/1500 top of podman

❯ podman run --rm -it 416d28a3d9116fb949081520fd8a1ee9c90ae9cc18b03a075265d584c1d7bdb0 bash
[root@c728374c06ef layers]# id
uid=0(root) gid=0(root) groups=0(root)

❯ podman run --rm -it -u 1500:1500 416d28a3d9116fb949081520fd8a1ee9c90ae9cc18b03a075265d584c1d7bdb0 bash
bash-4.4$ id
uid=1500(1500) gid=1500(1500) groups=1500(1500)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants