was cloud-lifesciences taken off gcr.io before the deadline?

[purajit]

https://cloud.google.com/life-sciences/docs/how-tos/variant-transforms says

Cloud Life Sciences is deprecated and will no longer be available on Google Cloud after July 8, 2025

However,

$ docker pull gcr.io/cloud-lifesciences/gcp-variant-transforms:latest
Error response from daemon: Head "https://gcr.io/v2/cloud-lifesciences/gcp-variant-transforms/manifests/latest": denied: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/cloud-lifesciences/locations/us/repositories/gcr.io" (or it may not exist)

While this is for the API, I would have assumed this would apply to the Docker
images as well. Was this taken down 6 months before the publicly shared deadline?
Was there an announcement we missed?

[bolianyin]

The old images have too many security vulnerabilities for public access. Do you think you can build the image yourself and host it privately in your project? If it does not work, we can try to think of other ways.

[pgrosu]

There seem to be two on DockerHub:

• https://hub.docker.com/r/samesense/gcp-variant-transforms

• https://hub.docker.com/r/avabhyankar2/gcp-variant-transforms

[purajit]

The old images have too many security vulnerabilities for public access. Do you think you can build the image yourself and host it privately in your project? If it does not work, we can try to think of other ways.

I know haha - I'd love to get rid of it, but one of the teams was using it with a plan to deprecate it
before the deadline; I would love for us to not depend on it (we're pretty much fully on python3.12,
even readying up for 3.13, except for this).

Sadly, there are too many images involved - the process within gcp-variant-transforms pulls and
runs other images that are also not available anymore, and can't be configured via variables either.
The entire cloud-lifesciences has been taken down of what I can tell.

There seem to be two on DockerHub

I did notice that, but I'm looking for official images, and there's also the handful of other images
needed.

[bolianyin]

Could you send the service accounts that need access to the images to [email protected]? We might be able to give access to some users, but not to the public.

[pgrosu]

Hi Bolian (@bolianyin),

I think Purajit (@purajit) is going to have trouble creating the Docker image using the provided Dockerfile, as it will produce the following errors:

Get:47 http://deb.debian.org/debian bullseye/main amd64 python3-wheel all 0.34.2-1 [24.0kB]
Get:48 http://deb.debian.org/debian bullseye/main amd64 python3-pip all 20.3.4-4+deb11u1[337 kB]
Get:49 http://deb.debian.org/debian bullseye/main amd64 python3-venv amd64 3.9.2-3 [1188B]
Fetched 138 MB in 3min 33s (648 kB/s)
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4%2bdeb11u4_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/perl_5.32.1-4%2bdeb11u4_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/p/perl/perl-modules-5.32_5.32.1-4%2bdeb11u4_all.deb  Connection timed out [IP: 151.101.2.132 80]
E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1w-0%2bdeb11u2_amd64.deb  Connection timed out [IP: 151.101.2.132 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get update && apt-get install -y     apt-transport-https    autoconf     automake     g++     gcc     gnupg     google-cloud-sdk     libbz2-dev     libcurl4-openssl-dev     liblzma-dev     libssl-dev     make     perl     zlib1g-dev     python3-pip     python3-venv' returned a non-zero code: 100
paul:~/tests/docker_tests/gcp-variant-transforms$

Basically, Debian 11 "Bullseye" reached the end of its regular support phase on August 14, 2024, and is now in LTS (Long Term Support). That would probably mean updating the repositories in the base image of the second stage (apache/beam_python3.8_sdk:2.37.0) to ensure newer versions are accessible, but that would also require regression testing of GCP Variant Transforms to make sure it is still giving the same results for all the tests.

Hope it helps,
~p

[purajit]

My teammate actually did manage to build an image (even before I filed this issue), but the problem
is that during the running process of gcp-variant-transforms, it pulls other cloud-lifesciences
images to run them, and they're all unavailable now, and there's no way to configure where those images
are fetched from, meaning we would not only have to build each image, but also make changes to internals
to be able to pull from our own container repo. That's when I filed the issue.

I believe the team is figuring out what service accounts we need for this and responding to that email.

[pgrosu]

Hi Purajit (@purajit),

That's cool that you tried that! So if we look back at the original error, it indicated a denied: Unauthenticated request...[for the path]...repositories/gcr.io response. So if you search the gcp-variant-transforms codebase, the cloud-lifesciences images are all part of Google's gcr.io repository. After you are given access for the service account, any launches under that account should just inherit those authenticated permissions for any program launches (giving you access to all those cloud-lifesciences images).

Hope it helps,
~p