Bitbucket "authorization denied" --> API call limitation? #882

AndyZe · 2024-09-18T19:30:18Z

On Sep 17 I started experiencing this issue on my bitbucket pipelines:

time="2024-09-18T19:00:50.149245108Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "

I guess it's probably not a coincidence that bitbucket deprecated legacy pipelines on Sep 17.

There are some solutions described here

Basically it sounds like this is deprecated: DOCKER_BUILDKIT=0

Maybe the new way to do it is: export PATH=/usr/bin:$PATH

Here's my bitbucket-pipelines.yml with the fix I attempted. It didn't work.

image: docker:git

pipelines:
  default:
     - step:
         services:
           - docker
         script:
           # Attempted fix
           # See https://community.atlassian.com/t5/Bitbucket-questions/Docker-build-failing-for-buildkit-with-error-authorization/qaq-p/2377667#M94515
           - export PATH=/usr/bin:$PATH
           - apk add --update bash coreutils tar
           - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
           - .industrial_ci/bitbucket.sh ROS_DISTRO=humble

definitions:
  services:
    docker:
      memory: 2048

The text was updated successfully, but these errors were encountered:

mathias-luedtke · 2024-09-19T05:56:34Z

Thanks for reporting!

time="2024-09-18T19:00:50.149245108Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "

Could you give me a little bit more context, please?
This looks like the "docker pull" fails or gets blocked.
You could set TRACE=true as well, which should print all the internal industrial_ci commands, which should help to narrow the scope.

Basically it sounds like this is deprecated: DOCKER_BUILDKIT=0

This has been deprecated (by docker) for a while now.
And it does not matter, because industrial_ci does not build any docker images. It just runs them.

Maybe the new way to do it is: export PATH=/usr/bin:$PATH

I would not mess with $PATH unless there is a good reason to do that.
To debug that more: Could you add an env call to your script and see what is the default for PATH.
And does bitbucket set any DOCKER_* environment variables?

mathias-luedtke · 2024-09-19T06:08:59Z

According to https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/ running docker containers should be a supported use case.
image: docker:git is not listed in any of the examples, though.

And it looks like bitbucket restricts the locations that can be mounted via volumes.
Please try setting DOCKER_CREDENTIALS="". This should disable the extra-mounts, but as well the option to clone other repos.

If Docker does work anymore, there is as well the ISOLATION=shell option:
https://github.com/ros-industrial/industrial_ci/blob/master/.gitlab-ci.yml#L39

AndyZe · 2024-09-19T15:23:05Z

I added this stuff to bitbucket-pipelines.yml. Didn't seem to change anything, though-

image: docker:git
pipelines:
  default:
     - step:
         size: 2x # More memory
         services:
           - docker
         script:
           - export TRACE=true
           - env
           - export DOCKER_CREDENTIALS=""
           - git clone [email protected]:apptronik/elevate_robotics.git
           - apk add --update bash coreutils tar
           - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
           - .industrial_ci/bitbucket.sh ROS_DISTRO=humble
definitions:
  services:
    docker:
      memory: 4096

-->

Build tab
...
Setting up ros-humble-ackermann-steering-controller (2.37.3-1jammy.20240911.165226) ...
Setting up ros-humble-tricycle-steering-controller (2.37.3-1jammy.20240911.165229) ...
Setting up ros-humble-bicycle-steering-controller (2.37.3-1jammy.20240911.165228) ...
Setting up ros-humble-ros2-controllers (2.37.3-1jammy.20240911.165933) ...
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
'install_target_dependencies' returned with code '0' after 1 min 49 sec
TRACE:util.sh:208 ici_hook after_install_target_dependencies

docker tab
...
time="2024-09-19T15:17:33.492890622Z" level=info msg="loading plugin "io.containerd.internal.v1.shutdown"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-09-19T15:17:33.492911474Z" level=info msg="loading plugin "io.containerd.ttrpc.v1.task"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-19T15:17:33.493167953Z" level=info msg="loading plugin "io.containerd.ttrpc.v1.pause"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-19T15:17:33.650782870Z" level=info msg="ignoring event" container=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2024-09-19T15:17:33.650808975Z" level=info msg="shim disconnected" id=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e namespace=moby
time="2024-09-19T15:17:33.650862345Z" level=warning msg="cleaning up after shim disconnected" id=5d9f648cf4e8cbd631df2e89534d58550cc785d2a7316d5f65c72b7b627f948e namespace=moby
time="2024-09-19T15:17:33.650873086Z" level=info msg="cleaning up dead shim" namespace=moby
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=false method=HEAD plugin=pipelines uri=/_ping
time="2024-09-19T15:17:33.857952559Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/json
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/attach?stderr=1&stdout=1&stream=1"
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/wait?condition=removed"
time="2024-09-19T15:17:33Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.44/containers/9319249432c6f9b7f2ed4e10cdead000a947c5b49a4c9347aa5266f19836c03b/start
time="2024-09-19T15:17:33.905484049Z" level=info msg="loading plugin "io.containerd.event.v1.publisher"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-09-19T15:17:33.905795247Z" level=info msg="loading plugin "io.containerd.internal.v1.shutdown"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-09-19T15:17:33.908529910Z" level=info msg="loading plugin "io.containerd.ttrpc.v1.task"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-19T15:17:33.908904756Z" level=info msg="loading plugin "io.containerd.ttrpc.v1.pause"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

AndyZe · 2024-09-19T15:36:54Z

Here's the env you asked for:

+ env
BITBUCKET_SSH_KEY_FILE=/opt/atlassian/pipelines/agent/ssh/id_rsa
BITBUCKET_REPO_UUID={413cc327-3443-45a1-bf0d-5a8f446b71c9}
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.34.224.1:443
CI=true
HOSTNAME=a70749b6-7bc4-4417-bf35-c6680ce44179-fk95q
BITBUCKET_STEP_TRIGGERER_UUID={d5ee4493-ec50-4736-919e-2b0c386ec051}
BITBUCKET_REPO_SLUG=aladdin_monitors
SHLVL=3
HOME=/root
[email protected]:apptronik/aladdin_monitors.git
BITBUCKET_REPO_OWNER=apptronik
BITBUCKET_STEP_UUID={a70749b6-7bc4-4417-bf35-c6680ce44179}
BITBUCKET_BUILD_NUMBER=13
BITBUCKET_WORKSPACE=apptronik
BITBUCKET_CLONE_DIR=/opt/atlassian/pipelines/agent/build
BITBUCKET_GIT_HTTP_ORIGIN=http://bitbucket.org/apptronik/aladdin_monitors
BITBUCKET_REPO_IS_PRIVATE=true
PIPELINES_JWT_TOKEN=$PIPELINES_JWT_TOKEN
NPM_CONFIG_USER=65534
BITBUCKET_COMMIT=ebdf04f32a398291df1f1899931aaf35236962e8
DIND_COMMIT=65cfcc28ab37cb75e1560e4b4738719c07c6618e
BITBUCKET_PIPELINE_UUID={1316958a-3471-4095-b122-7802c08085d5}
BITBUCKET_DOCKER_HOST_INTERNAL=10.38.175.11
KUBERNETES_PORT_443_TCP_ADDR=10.34.224.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
BITBUCKET_REPO_OWNER_UUID={d5a60e11-1f8d-4fc5-adbf-f55f315a1124}
KUBERNETES_PORT_443_TCP_PROTO=tcp
BITBUCKET_STEP_RUN_NUMBER=1
BITBUCKET_BRANCH=main
DOCKER_VERSION=25.0.5
DOCKER_TLS_CERTDIR=/certs
BITBUCKET_REPO_FULL_NAME=apptronik/aladdin_monitors
DOCKER_HOST=tcp://localhost:2375
BITBUCKET_PROJECT_UUID={3462a40f-cc88-4642-86e5-cb294a61ebf7}
BITBUCKET_PROJECT_KEY=SCOR
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.34.224.1:443
DOCKER_BUILDX_VERSION=0.13.1
DOCKER_COMPOSE_VERSION=2.25.0
KUBERNETES_SERVICE_HOST=10.34.224.1
PWD=/opt/atlassian/pipelines/agent/build
TRACE=true

AndyZe · 2024-09-19T15:59:12Z

This may all be a red herring. I have other bitbucket pipelines that are still succeeding, same bitbucket-pipelines.yml. Very confused...

mathias-luedtke · 2024-09-20T18:24:49Z

According to your logs, industrial_ci is running and even managed to start the docker container and install things.
What happens, if you remove "image: docker:git"?

AndyZe · 2024-09-21T20:41:49Z

Same symptoms unfortunately. The build tab is stuck at...

install_target_dependencies
...
Setting up ros-humble-moveit-resources-panda-moveit-config (2.0.7-1jammy.20240830.212942) ...
Setting up adwaita-icon-theme (41.0-1ubuntu1) ...
Setting up humanity-icon-theme (0.6.16) ...
Setting up ubuntu-mono (20.10-0ubuntu2) ...
Setting up libgtk-3-0:amd64 (3.24.33-1ubuntu2.2) ...
Setting up libgtk-3-bin (3.24.33-1ubuntu2.2) ...
Setting up qt5-gtk-platformtheme:amd64 (5.15.3+dfsg-2ubuntu0.2) ...

The docker tab is stuck at... (don't ask me why there's still a docker tab)

time="2024-09-21T16:47:04Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.41/containers/2ca37a1066c1860ebd3ca85ec48b165e2943bfb1f3db83ee87ae710ec7a7dd84/start
time="2024-09-21T16:47:04.711954506Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-09-21T16:47:04.712371367Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-09-21T16:47:04.712393985Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-21T16:47:04.712543646Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

AndyZe · 2024-09-21T20:57:43Z

Maybe relevant:

I guess there is some sort of rate limiting mechanism in Bitbucket pipelines causing only the first few pulls to succeed from ecr. I solved the issue by starting SAM with --skip-pull-image option.

https://stackoverflow.com/q/78424795
https://support.atlassian.com/bitbucket-cloud/docs/api-request-limits/

Is the docker image that industrial_ci uses based on ros-desktop-full? Maybe that would help?

AndyZe · 2024-09-21T22:36:30Z

OK, if I remove some deps, I can at least get to the point where the tests are run. They don't pass because controller_manager package is missing but that verifies that it's a bitbucket API call limitation.

mathias-luedtke · 2024-09-22T11:53:15Z

OK, if I remove some deps, I can at least get to the point where the tests are run. They don't pass because controller_manager package is missing but that verifies that it's a bitbucket API call limitation.

For me it looks like 2h time limit for the pipeline.
(https://community.atlassian.com/t5/Bitbucket-questions/Increasing-pipeline-runner-time-limits/qaq-p/2013093)
industial_ci should only pull the image once and then start it.
How long does it take locally? (https://github.com/ros-industrial/industrial_ci/blob/master/doc/index.rst#run-industrial-ci-on-local-host)

Is the docker image that industrial_ci uses based on ros-desktop-full? Maybe that would help?

No, industrial_ci uses ubuntu:$codename by default.
You can override it with ´DOCKER_IMAGE'.

AndyZe · 2024-09-22T12:50:02Z

It only takes about 30s to build and another 30s to run tests locally. It's not a huge pkg at all but it does have a lot of dependencies (moveit2 & ros2_control).

OK, this got me to the point where CI quickly runs tests. I'm happy with that! Closing the issue, thanks for the hints.

         script:
           - export DOCKER_IMAGE="ros:humble"

I think what's going on is it hits the bitbucket API limit then just spins until it times out.

AndyZe closed this as completed Sep 19, 2024

AndyZe reopened this Sep 19, 2024

AndyZe changed the title ~~Bitbucket legacy pipelines deprecated --> "authorization denied"~~ Bitbucket "authorization denied" --> API call limitation Sep 21, 2024

AndyZe changed the title ~~Bitbucket "authorization denied" --> API call limitation~~ Bitbucket "authorization denied" --> API call limitation? Sep 22, 2024

AndyZe closed this as completed Sep 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bitbucket "authorization denied" --> API call limitation? #882

Bitbucket "authorization denied" --> API call limitation? #882

AndyZe commented Sep 18, 2024

mathias-luedtke commented Sep 19, 2024

mathias-luedtke commented Sep 19, 2024

AndyZe commented Sep 19, 2024 •

edited

Loading

AndyZe commented Sep 19, 2024

AndyZe commented Sep 19, 2024

mathias-luedtke commented Sep 20, 2024

AndyZe commented Sep 21, 2024

AndyZe commented Sep 21, 2024 •

edited

Loading

AndyZe commented Sep 21, 2024

mathias-luedtke commented Sep 22, 2024

AndyZe commented Sep 22, 2024 •

edited

Loading

Bitbucket "authorization denied" --> API call limitation? #882

Bitbucket "authorization denied" --> API call limitation? #882

Comments

AndyZe commented Sep 18, 2024

mathias-luedtke commented Sep 19, 2024

mathias-luedtke commented Sep 19, 2024

AndyZe commented Sep 19, 2024 • edited Loading

AndyZe commented Sep 19, 2024

AndyZe commented Sep 19, 2024

mathias-luedtke commented Sep 20, 2024

AndyZe commented Sep 21, 2024

AndyZe commented Sep 21, 2024 • edited Loading

AndyZe commented Sep 21, 2024

mathias-luedtke commented Sep 22, 2024

AndyZe commented Sep 22, 2024 • edited Loading

AndyZe commented Sep 19, 2024 •

edited

Loading

AndyZe commented Sep 21, 2024 •

edited

Loading

AndyZe commented Sep 22, 2024 •

edited

Loading