Helm pull from ACR: 401 unauthorized with kubelet managed identity

[adeturner]

Hi,

Please can someone help explain a way forward for me? Source controller is getting 401 unauth pulling a helm chart from ACR, but AKS is setup with kubelet managed identity and AcrPull granted, proven by the check-acr command below. The docs say just kubelet permissions should be enough.

$ k logs -n flux-system source-controller-69bf47ff7d-gg2hv | grep myartifact | tail -1
{"level":"error","ts":"2023-07-12T07:08:46.152Z","msg":"Reconciler error","controller":"helmchart",
"controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart",
"HelmChart":{"name":"myartifact-myartifact","namespace":"myartifact"},
"namespace":"myartifact","name":"myartifact-myartifact",
"reconcileID":"d7817cf9-6f38-4d61-b28f-ddef54cc7048",
"error":"chart pull error: failed to get chart version for remote reference: could not get tags for \"myartifact-chart\": 
could not fetch tags for \"oci://myacr.azurecr.io/helm/myartifact-chart\": 
GET \"https://myacr.azurecr.io/v2/helm/myartifact-chart/tags/list\": 
GET \"https://myacr.azurecr.io/oauth2/token?scope=repository%3Ahelm%2Fmyartifact-chart%3Ametadata_read%2Cpull&service=myacr.azurecr.io\": 
unexpected status code 401: unauthorized: 
authentication required, visit https://aka.ms/acr/authorization for more information."}

$ az aks check-acr --resource-group $RG --name $CLUSTER_NAME --acr myacr.azurecr.io

[2023-07-12T07:04:17Z] Checking host name resolution (myacr.azurecr.io): SUCCEEDED
[2023-07-12T07:04:17Z] Canonical name for ACR (myacr.azurecr.io): FFFFFFFFFF.westeurope.cloudapp.azure.com.
[2023-07-12T07:04:17Z] ACR location: westeurope
[2023-07-12T07:04:17Z] Checking managed identity...
[2023-07-12T07:04:17Z] Kubelet managed identity client ID: FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF
[2023-07-12T07:04:17Z] Validating managed identity existance: SUCCEEDED
[2023-07-12T07:04:17Z] Validating image pull permission: SUCCEEDED
[2023-07-12T07:04:17Z]
Your cluster can pull images from myacr.azurecr.io!

$ f check
► checking prerequisites
✔ Kubernetes 1.25.6 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.34.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.4
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.0.0-rc.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.0.0-rc.5
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

[somtochiama]

Hello. Did you set .spec.provider: azure in your manifests or does your cluster have more than one identity attached?

[hiddeco]

The following documentation should help you with any further questions: https://fluxcd.io/flux/components/source/helmrepositories/#azure

[adeturner]

Thanks @hiddeco, with respect, I'm not sure that documentation is clear enough - to me it reads that kubelet identity should be enough.
I just found this issue which says if AAD pod identity is present (it is in my cluster) the call to "DefaultAzureCredential" does not default to Kubelet Identity first. I'll try configuring AAD Pod Identity, but I'm actually trying to remove it.

[adeturner]

Thanks for the help, I managed to pull the chart. Using the workload identity section I configured source-controller patches, then added a federated identity to the kubelet managed identity. I did not have to install a workload identity mutating webhook.

From my perspective, the relationship and dependencies between the helmrepo doc sections are not clear.

• Its a bit strange the opening paragraph avoids mentioning workload identity.
• I cant see from the text that kubelet managed identity is only a partial solution.
• Its not clear that I had to specify provider: azure in the helmrepository (although obvious retrospectively).
• I did not have to install a workload identity mutating webhook (I could not when I tried, is this is pre-installed now?)

I can raise a bug if you agree.

[somtochiama]

@adeturner I agree that the docs could use some updates.

kubelet managed identity is only a partial solution.

Kubelet-managed identity isn't really a partial solution. You start encountering issues when you have multiple identities or other authentication methods configured. However, enough people are running in this that we should call it out in the docs

did not have to install a workload identity mutating webhook (I could not when I tried, is this pre-installed now?)

Passing a flag to the azure cli now configures this for you. Thanks for pointing this out

Its not clear that I had to specify provider: azure in the helmrepository

This is mentioned in the docs. I think the problem is that people sometimes skim through and miss it

[boerngen-schmidt]

@adeturner Coudl you please share your configuration on how you got the workload identity working with ACR. I'm currently stuck on how to apply the service account. patch mentioned in the documentation

[boerngen-schmidt]

Hello,

I tried using the Kublet Identity to authenticate against ACR and one major configuration I was missing was the activation for Microsoft GitOps to use the KubeletIdentity. Found it mentioned here and later on MS

az k8s-extension update -c $CLUSTER -g $RG -n flux -t ManagedClusters --config =true

So for reference the full configuration would look similar to this:

# Create Identities
az identity create --name aks_identity -g $RG
az identity create --name kubelet_identity -g $RG
aksId=$(az identity show -g $RG -n aks_identity --query id -o tsv)
kubletId=$(az identity show -g $RG -n kubelet_identity --query id -o tsv)
# Update Cluster to use managed identities
az aks update -g $RG -n $CLUSTER \
  --enable-managed-identity \
  --assign-identity $aksId \
  --assign-kubelet-identity $kubletId
# Assign Rights to ACR for kubelet Identity
kubeletSpId=$(az identity show -g $RG -n kubelet_identity --query principalId --output tsv)
acrResourceId=$(az acr show -g $RG -n $ACR --query id --output tsv)
az role assignment create --assignee $kubeletSpId --scope $acrResourceId --role acrpull
# Update GitOps extension to use kublet identity
az k8s-extension update -c $CLUSTER -g $RG -n flux -t ManagedClusters --config useKubeletIdentity=true