Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploying determined ai on premise kubernetes cluster with custom registry for the postgres db pod results in forbidden error while pulling image🐛[bug] #7770

Open
humbleearth opened this issue Aug 31, 2023 · 3 comments
Labels

Comments

@humbleearth
Copy link

Describe the bug

The postgres db pod image does not download with custom registry.

Reproduction Steps

  1. Download the latest helm code
  2. Update values.yaml file with imagePullSecretName value and defaultImages:
    imageRegistry: sample.registry.custom.com:5000/determinedai

PostgreSQL image

postgreSQL: "sample.registry.custom.com:5000/postgres:10.14"
4. helm install

The master image downloads but while downloading the postgresql image results in below error:

Error inside db pod:
Warning Failed 6s (x3 over 46s) kubelet Failed to pull image
"sample.registry.custom.com:5000/postgres:10.14": rpc error: code = Unknown desc = failed to pull and unpack image "sample.registry.custom.com:5000/postgres:10.14": failed to resolve reference "sample.registry.custom.com:5000/postgres:10.14": pulling from hostsample.registry.custom.com:5000 failed with status code [manifests
10.14]: 403 Forbidden

Expected Behavior

It should pull both the master and postgres images

Screenshot

image

Environment

  • Device or hardware: k8s
  • OS: linux
  • Browser n/a
  • Version 0.24.0

Additional Context

No response

@ioga
Copy link
Contributor

ioga commented Aug 31, 2023

Hello,

Sorry about that. postgres is a public image, it's uncommon to use a private registry for it and we've never hit this before.

As a workaround, you can change the db deployment to set imagePullSecrets the same way master deployment does it.

@tpanza
Copy link

tpanza commented Aug 31, 2023

hi @ioga . In a corporate environment, we have to route all container image requests through an on-prem Artifactory server that sits between the corporate network and the outside world. This includes even public images like postgres

Could you please consider updating the helm chart to provide a "global" imagePullSecret that is used for all image pulls?

Further, for all of the images (master, postgreSQL, kubeScheduler, kubeSchedulerPreemption, cpuImage, etc), please consider having the Helm chart break that up into 3 separate editable fields: registry, repository, tag.

Then in this scenario, we would only override the registry for each image to point to the Artifactory proxy server, and leave the repository and tag as defaults

@ioga
Copy link
Contributor

ioga commented Aug 31, 2023

Your request makes sense to me. I've added an internal tickets to address these two points.

if you end up forking the helm chart and implementing this, while keeping the backwards compatibility with the existing defaultImages & imageRegistry options, we'd be happy to take a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants