✨ Adding support for multiple registry mirrors in bottlerocket

[pokearu]

What this PR does / why we need it:
Currently the registry mirror configuration for Bottlerocket was hardcoded to setup mirrors for only public.ecr.aws. With these changes we introduce new API fields for RegistryMirrorConfiguration that enable listing multiple registries to be mirrored to a given private registry.

Example

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      registryMirror:
        caCert: |
          -----BEGIN CERTIFICATE-----
          MIIF5DCCA8ygAwIBAgIUXgwAZqssgRQ/Kd3HKNJC2sTVsWQwDQYJKoZIhvcNAQEL
          ...
          -----END CERTIFICATE-----
        mirrors:
          - registry: "public.ecr.aws"
            endpoint: "1.2.3.4:443/eks-a"
          - registry: "docker.io"
            endpoint: "1.2.3.4:443"

The above configuration would setup bottlerocket settings through the user data and setup mirrors as below

{
  "settings": {
    "container-registry": {
      "mirrors": [
        {
          "endpoint": [
            "1.2.3.4:443/eks-a"
          ],
          "registry": "public.ecr.aws"
        },
        {
          "endpoint": [
            "1.2.3.4:443"
          ],
          "registry": "docker.io"
        }
      ]
    }
  }
}

If the RegistryMirrorConfiguration.Endpoint field is set and no mirrors are found, we still default to setting up public.ecr.aws as the registry.

[jiayiwang7]

// We need to assign creds for "public.ecr.aws" because host-ctr expects credentials to be assigned
// to "public.ecr.aws" rather than the mirror's endpoint
// TODO: Once the bottlerocket fixes are in we need to remove the "public.ecr.aws" creds

Do we validate that this is fixed by BR? is it still required to assign creds to the public.ecr.aws registry?
Also looks like you are adding credentials for all the mirrors here, is it necessary?

[pokearu]

So I discussed this with the team. Looks like the issue is still open bottlerocket-os/bottlerocket#2677

Regarding setting creds for all reg, I am not sure if its actually necessary but we suspected it might need it. I can check again 👍

[pokearu]

Just confirmed. It only needs public ecr as an extra entry for creds. I can change that up.

[jiayiwang7]

Mind adding detailed explanation about why we are doing this? e.g. for backward compatibility, not trigger rollout recreate of the machine etc.

[jiayiwang7]

do we need bottlerocketInput.RegistryMirrorEndpoint at all? won't it always be config.RegistryMirrorConfiguration.Mirrors[0].Endpoints[0]?

[pokearu]

No it wont be. So in this case we are using the older api and not setting mirrors at all. This was mainly done for backward compatibility and not rolling.
Example

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      registryMirror:
        caCert: |
          -----BEGIN CERTIFICATE-----
          MIIF5DCCA8ygAwIBAgIUXgwAZqssgRQ/Kd3HKNJC2sTVsWQwDQYJKoZIhvcNAQEL
          ...
          -----END CERTIFICATE-----
          endpoint: 1.2.3.4:443

Is still a valid way of setting it up.

[pokearu]

I think i got what you were asking now. You meant bottlerocketInput.RegistryMirrorEndpoint not the config.RegistryMirrorConfiguration.Endpoint.

I just used it so its easier for go template parsing.

[jiayiwang7]

yea i mean you don't need to define bottlerocketInput.RegistryMirrorEndpoint, the registry map is enough to build the template.

[jiayiwang7]

i think we need to call this out in API definition

[pokearu]

sure!

[jiayiwang7]

this can return nil, can we handle it?