Error updating APIService "v3.projectcalico.org"

[kanyi47]

ENVIRONMENT:
DigitalOceans Droplets

OS:
Ubuntu 22.04 (LTS) X64
KUBERNETES 1.29.1
CONTAINERD 1.7.13
RUNC 1.1.12
CNI PLUGINS 1.4.0
CALICO CNI 3.27.2

I've had everything set up and running correctly for almost a week now, including load balancer and domain. But, out of nowhere, I start getting response "503 Service Unavailable" when I search my domain name. All pods are in running and nodes are in Ready state.
I had this issue a while ago and it resolved itself after a few days. Then it happened again, and I resolved by reinstalling Calico. A check of kube-apiserver-k8s-control logs points to v3.projectcalico.org error. Here are some of the logs:

I1130 07:50:26.822732 1 controller.go:624] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io I1130 07:50:26.977203 1 handler.go:275] Adding GroupVersion projectcalico.org v3 to ResourceManager W1130 07:50:26.987509 1 handler_proxy.go:93] no RequestInfo found in the context E1130 07:50:26.987585 1 controller.go:146] Error updating APIService "v3.projectcalico.org" with err: failed to download v3.projectcalico.org: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]] E1130 07:50:26.987950 1 handler_proxy.go:137] error resolving calico-apiserver/calico-api: service "calico-api" not found I1130 07:50:27.071858 1 alloc.go:330] "allocated clusterIPs" service="calico-apiserver/calico-api" clusterIPs={"IPv4":"10.98.98.211"} W1130 07:50:27.100290 1 handler_proxy.go:93] no RequestInfo found in the context E1130 07:50:27.102307 1 controller.go:146] Error updating APIService "v3.projectcalico.org" with err: failed to download v3.projectcalico.org: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

The "calico-api" not found log is quite confusing because I do have it, and it's correctly set up, I think:
root@master-node:~# kubectl get apiservice v3.projectcalico.org NAME SERVICE AVAILABLE AGE v3.projectcalico.org calico-apiserver/calico-api True 10d

I tried restarting kube-apiserver-k8s control pod, and it did restart but generated more logs related to v3.projectcalico.org. Here are some of the significant ones:
2024/12/10 16:36:53 httputil: ReverseProxy read error during body copy: unexpected EOF 2024/12/10 16:36:53 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:36:53.995113 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/felixconfigurations?allowWatchBookmarks=true&resourceVersion=10768949&timeout=9m25s&timeoutSeconds=565&watch=true" audit-ID="74d845f3-5338-4361-8e08-ed64dd37fa3a" E1210 16:36:53.995438 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/networksets?allowWatchBookmarks=true&resourceVersion=10768957&timeout=5m40s&timeoutSeconds=340&watch=true" audit-ID="eab87f6d-bd62-4448-a51c-9f623a6368fd" E1210 16:36:53.995665 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/kubecontrollersconfigurations?allowWatchBookmarks=true&resourceVersion=10768949&timeout=9m35s&timeoutSeconds=575&watch=true" audit-ID="f21f46cc-2207-4994-b77b-4e60f1e12ae9" E1210 16:36:53.996987 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/bgppeers?allowWatchBookmarks=true&resourceVersion=10768953&timeout=9m46s&timeoutSeconds=586&watch=true" audit-ID="6af32e45-6f3b-47bb-b873-82ec13fafb8e" 2024/12/10 16:37:21 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:37:21.073867 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/networksets?allowWatchBookmarks=true&resourceVersion=12856876&timeout=6m52s&timeoutSeconds=412&watch=true" audit-ID="76dca997-e9ed-44d1-825b-99a8293254f9" 2024/12/10 16:37:21 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:37:21.074511 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/ipamconfigurations?allowWatchBookmarks=true&resourceVersion=12856876&timeoutSeconds=523&watch=true" audit-ID="3e15a46b-79c3-4633-af34-dd18afc10def" 2024/12/10 16:37:21 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:37:21.074712 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/ippools?allowWatchBookmarks=true&resourceVersion=12856871&timeout=8m30s&timeoutSeconds=510&watch=true" audit-ID="fd0956b9-39f9-48f8-a5a4-713aa584eb90" 2024/12/10 16:37:21 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:37:21.074942 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/bgppeers?allowWatchBookmarks=true&resourceVersion=12856872&timeout=6m47s&timeoutSeconds=407&watch=true" audit-ID="6e091af3-441a-4263-9a03-804b2de8ec24" 2024/12/10 16:37:21 httputil: ReverseProxy read error during body copy: unexpected EOF E1210 16:37:21.085132 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/projectcalico.org/v3/globalnetworksets?allowWatchBookmarks=true&resourceVersion=12856875&timeout=8m39s&timeoutSeconds=519&watch=true" audit-ID="17f7e184-f8b8-4664-8cc7-d472c1174f7b" W1210 16:37:25.958986 1 handler_proxy.go:93] no RequestInfo found in the context E1210 16:37:25.959119 1 controller.go:146] Error updating APIService "v3.projectcalico.org" with err: failed to download v3.projectcalico.org: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]] E1210 16:37:25.960645 1 available_controller.go:460] v3.projectcalico.org failed with: failing or missing response from https://10.97.99.201:443/apis/projectcalico.org/v3: Get "https://10.97.99.201:443/apis/projectcalico.org/v3": dial tcp 10.97.99.201:443: connect: connection refused

I'm not sure if it's a bug, but I've tried looking for maybe people who've experienced a similar issue, I can't find any. Please help. I'm literally stuck now, and I can't keep on reinstalling Calico every time it happens. Thanks.

[caseydavenport]

Sounds like a connectivity problem between the main Kubernetes API server and the Calico API server. Probably want to start there for diagnosis.

A few thoughts:

• Are other pod / nodes able to ping the IP address of the Calico API server?
• Is the Calico API server pod running?
• What do the logs say from the Calico API server pod?

[kanyi47]

@caseydavenport, it just solved itself again. I don't understand why this is happening. I need to figure this out so I make sure it doesn't happen again. About the suggestions you recommended, Calico API server pods were running and had no log errors, both of them. I wasn't able to ping the IP address while the issue was occurring though. Could you have any thoughts on why this is happening. Thanks.

[caseydavenport]

I wasn't able to ping the IP address while the issue was occurring though

Sounds likely that this is related to a transient underlying networking issue of some sort, if the pod IP addresses aren't reachable during that time (and the pods are running). Just to double check - did you ping the pod IP or the Service IP? The Service IP will never work for ICMP traffic.

[kanyi47]

did you ping the pod IP or the Service IP? The Service IP will never work for ICMP traffic.

I didn't manage to ping either while the issue was happening, but right now the pod IPs are reachable. The Service IP itself is not reachable, just as you said. Here are the results:

NAME                               READY   STATUS    RESTARTS      AGE     IP              NODE          NOMINATED NODE   READINESS GATES
calico-apiserver-fcb88dcd4-ngwkz   1/1     Running   1 (15m ago)   2d10h   10.244.175.3    k8s-control   <none>           <none>
calico-apiserver-fcb88dcd4-pp8hg   1/1     Running   0             2d10h   10.244.171.44   worker-01     <none>           <none>

root@master-node:~# ping 10.244.175.3 
PING 10.244.175.3 (10.244.175.3) 56(84) bytes of data.
64 bytes from 10.244.175.3: icmp_seq=1 ttl=64 time=0.102 ms
---
--- 10.244.175.3 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6121ms

root@worker-01:~# ping 10.244.171.44 
PING 10.244.171.44 (10.244.171.44) 56(84) bytes of data.
64 bytes from 10.244.171.44: icmp_seq=1 ttl=64 time=0.060 ms
---
--- 10.244.171.44 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5104ms