Struggling to pull images from spegel

[plauche]

Hello!

I recently read about the embedded registry mirror in k3s and I wanted to see if I could use this as a local registry for manually imported images. I'm just running a single node cluster, so I don't actually need mirroring, but I want the ability to manually load images at will. This appears to be possible according to the docs, but I'm struggling to get k3s to pull images from spegel instead of docker.io.

I'm currently running k3s version v1.30.3+k3s1 with the --embedded-registry flag. (/usr/local/bin/k3s server --embedded-registry).

My /etc/rancher/k3s/registries.yaml:

mirrors:
  "*":
configs:
  "*":
    tls:
      insecure_skip_verify: true

Image loaded with ctr -n=k8s.io (sudo ctr -n=k8s.io image ls):

REF            TYPE                                       DIGEST                                                                  SIZE      PLATFORMS   LABELS 
api-server:1.0 application/vnd.oci.image.manifest.v1+json sha256:7f1a8d5e937e65379497d711dc43327179ddb5cd7e6fa3c91a32de0813c26391 144.8 MiB linux/amd64 -

Deployment file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: api-server
  labels:
    app: api-server
spec:
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - image: api-server:1.0
          name: api-server
          command: ["api-server"]
          args: ["-p", "5555"]
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "8Mi"
              cpu: "100m"
      restartPolicy: Always
      hostNetwork: true

Output from kubectl describe pods:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  19s               default-scheduler  Successfully assigned default/api-server-g8q6d to plauche-cw-pad
  Normal   BackOff    17s               kubelet            Back-off pulling image "api-server:1.0"
  Warning  Failed     17s               kubelet            Error: ImagePullBackOff
  Normal   Pulling    3s (x2 over 18s)  kubelet            Pulling image "api-server:1.0"
  Warning  Failed     3s (x2 over 18s)  kubelet            Failed to pull image "api-server:1.0": failed to pull and unpack image "docker.io/library/api-server:1.0": failed to resolve reference "docker.io/library/api-server:1.0": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     3s (x2 over 18s)  kubelet            Error: ErrImagePull

Output from kubectl get --raw /api/v1/nodes/blah/proxy/metrics | grep -F spegel:

# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="127.0.0.1:5000"} 1
spegel_advertised_image_digests{registry="docker.io"} 9
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="127.0.0.1:5000"} 1
spegel_advertised_image_tags{registry="docker.io"} 9
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="127.0.0.1:5000"} 2
spegel_advertised_images{registry="docker.io"} 18
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="127.0.0.1:5000"} 6
spegel_advertised_keys{registry="docker.io"} 69
# HELP spegel_mirror_requests_total Total number of mirror requests.
# TYPE spegel_mirror_requests_total counter
spegel_mirror_requests_total{cache="miss",registry="docker.io",source="internal"} 6
# HELP spegel_resolve_duration_seconds The duration for router to resolve a peer.
# TYPE spegel_resolve_duration_seconds histogram
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.005"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.01"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.025"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.05"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.1"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.25"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="0.5"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="1"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="2.5"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="5"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="10"} 0
spegel_resolve_duration_seconds_bucket{router="libp2p",le="+Inf"} 0
spegel_resolve_duration_seconds_sum{router="libp2p"} 0
spegel_resolve_duration_seconds_count{router="libp2p"} 0

The containerd logs seem to indicate it looking at spegel (port 6443) but isn't able to find it...

time="2024-08-20T17:13:52.527271126-05:00" level=info msg="PullImage \"api-server:1.0\""
time="2024-08-20T17:13:52.537937100-05:00" level=info msg="trying next host - response was http.StatusNotFound" host="127.0.0.1:6443"
time="2024-08-20T17:13:52.893555588-05:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
time="2024-08-20T17:13:52.895355034-05:00" level=error msg="PullImage \"api-server:1.0\" failed" error="failed to pull and unpack image \"docker.io/library/api-server:1.0\": failed to resolve reference \"docker.io/library/api-server:1.0\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"

Any pointers on what configuration might be wrong would be so appreciated!

[brandond]

I want the ability to manually load images at will

You can already do this. Remember that Spegel has no image store, it just shares whatever is in containerd on each node. Spegel won't pull images from itself on a single node cluster. That doesn't make any sense as a thing to do - it would just be containerd loading from itself, and if it already has an image, it doesn't need to pull it.

If you want to load images into containerd, just use ctr to do so - you don't need to get spegel involved.

[plauche]

Thanks for that information! I think that clarifies things a bit for me, but also made me realize what my real question is.

The thing I want to accomplish is creating a single node k3s deployment which uses an image from a local registry. I thought that after loading an image into containerd using the correct namespace (ctr -n k8s.io), that then spegel could act as a registry that k3s could pull from when deploying. Is that an incorrect way to think about how these pieces interact?

[plauche]

Ok I've resolved my own problem, but this answer was helpful in getting me thinking in a better direction! Thank you!!

Other issues I had:

• I was using my host ctr instead of k3s ctr
• I needed to fix the naming of my image: api-server:1.0 -> blah.com/api-server:1.0.