Kustomization gets "failed to download artifact .... dial tcp 10.100.32.193:80: i/o timeout" on eks

[shalomy-cyera]

Hi,

My cluster worked ok and nothing has changed, but since this morning I get from the kustomization this error:

Warning  error   32s (x2 over 9m52s)  kustomize-controller  failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/some_hash.tar.gz giving up after 10 attem │
│ pt(s): Get "http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/some_hashtar.gz": dial tcp 10.100.32.193:80: i/o timeout

We are running on EKS (v1.21.5-eks-bc4871b) and the flux version is 0.25.3.

flux version
flux: v0.25.3
helm-controller: v0.15.0
kustomize-controller: v0.19.1
notification-controller: v0.20.1
source-controller: v0.20.1

flux check
► checking prerequisites
✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.15.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.19.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.20.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.20.1
✔ all checks passed

Any idea what else can I do?
I installed new cluster and still happens.

[stefanprodan]

Looks like the EKS node, where kustomize-controller is running, has its network broken. This can happen due to CNI misconfiguration, CNI crash loop or network policies bugs in your CNI implementation.

[yershalom]

Thanks for the quick answer @stefanprodan , nothing has changed though but I'll check this way.
Thanks again

[stefanprodan]

dial tcp 10.100.32.193:80: i/o timeout means that pod to pod communication is broken, we can't do anything about it in Flux. Please reach out to AWS Support.

[sgolod]

This can happen when source-controller pod is on highly overloaded nodes.

[mehemken]

I'm getting the same error on a brand new EKS cluster.

❯ flux get kustomizations -A                                                                                                                                             master !
NAMESPACE       NAME            READY   MESSAGE

                                                                                        REVISION        SUSPENDED
flux-system     flux-system     False   failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/138a
cf6fcad20b0888de1352d471e00310f8f4d7.tar.gz giving up after 10 attempt(s): Get "http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/138a
cf6fcad20b0888de1352d471e00310f8f4d7.tar.gz": dial tcp 172.20.194.2:80: i/o timeout                     True

The same flux bootstrap command was working last weekend. It is not working today. I haven't touched the AWS VPC CNI settings. I've added extra security group rules to allow all TCP between nodes and the control plane. I've been hacking at this for over 4 hours and I can't get it to work. Last week I got it running immediately.

I'm getting this output from a pod within the flux-system namespace:

bash-5.1# curl -vvv http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/138acf6fcad20b0888de1352d471e00310f8f4d7.tar.gz --output foo.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.20.194.2:80...
* Connected to source-controller.flux-system.svc.cluster.local (172.20.194.2) port 80 (#0)
> GET /gitrepository/flux-system/flux-system/138acf6fcad20b0888de1352d471e00310f8f4d7.tar.gz HTTP/1.1
> Host: source-controller.flux-system.svc.cluster.local                                                                                                           �
> User-Agent: curl/7.80.0                                                                                                                                         �
> Accept: */*                                                                                                                                                     �
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 32850
< Content-Type: application/x-gzip
< Last-Modified: Sat, 26 Feb 2022 18:57:48 GMT
< Date: Sat, 26 Feb 2022 20:05:40 GMT
<
{ [512 bytes data]
100 32850  100 32850    0     0  19.8M      0 --:--:-- --:--:-- --:--:-- 31.3M
* Connection #0 to host source-controller.flux-system.svc.cluster.local left intact

... which leads me to believe that pod connectivity is fine.

[Smana]

Have you found a solution @mehemken ? I'm getting the same issue, and it was working perfectly yesterday.

[mehemken]

Hey @Smana , I ended up using different clusters. But looking at this now... maybe I didn't have the SSH key configured correctly. I can no longer confirm, but everything on my new clusters has been working correctly.

[Smana]

From the tests I just made, this seems to be caused by the network policies installed by Flux. I had to put an extra rule in the past in order to allow webhooks to work so I suspected this.
Deleting all network policies in the namespace flux-system confirmed that.
Now I need to find out the proper configuration.

[stefanprodan]

@joaqquin89 my guess is that your EKS cluster is broken, is CoreDNS running, did you installed a CNI on top of AWS’ one? Does the CNI allow traffic between pods?

[joaqquin89]

@stefanprodan we are using VPC CNI https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html. and we have installed instil too

[Smana]

Just to let you know that disabling the network policies at flux bootstrap fixed our issue

[joaqquin89]

@Smana , yes I did that thing in my terraform file .. but the problem still continue

[joaqquin89]

@Smana , @stefanprodan , guys .. I noticed the next thing .. I tried to to the tshoot (network tshoot) and I noticed that exists a lot of TIME_WAIT records and that private ip is the worker node private ip that contains this pod.

so I have an big doubt the worker node needs to connect to the pod in that port (9440) to do something special ?

bests

[vipulJain05]

Methods to solve this issue:

1. Check the security associated with the EKS cluster and check if that one has its security group id attached with the type of "All traffic", follow this article for reference https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html

2. Check the IP which the source controller is trying to connect in services of k8s like 10.100.32.193, it may give you an idea of which service is creating the issue

[joaqquin89]

Guys

in my case I solved this issue .. enabling self rules for the security group . im using the eks terraform module to bring up an k8s cluster
bellow , you can see the rule that fixed this problem in case

  node_security_group_additional_rules = {
     ingress_self_all = {
       description = "Node to node all ports/protocols"
       protocol    = "-1"
       from_port   = 0
       to_port     = 0
       type        = "ingress"
       self        = true
     }
   }

in case that you bring up an eks cluster using another tools(eksctl or manual) , try to add this rule. The aws CNI is not the problem

[klml]

Dot at the end of the domain

I had the same error using OpenShift:

failed to download archive, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/capc/e93d1865923776e61abb4e0fc962f0243b8908bd.tar.gz giving up after 10 attempt(s)"}

We have configured a cluster proxy with a noProxy: for our internal domain (using example.org here). This proxy is set as env in all deployments.
Openshift expands this with some IP ranges and internal domains:

status:
  httpProxy: http://10.10.0.10:80
  httpsProxy: http://10.10.0.10:80
  noProxy: .cluster.local,.example.org,.svc,10.167.0.0/16,10.168.0.0/16,10.145.0.31,10.145.30.0/24,127.0.0.1,api-int.friedolin.example.org,localhost  

So there is only .cluster.local but not .cluster.local.

After add this dot-ended domain, flux works.

spec:
  httpProxy: http://10.10.0.10:80
  httpsProxy: http://10.10.0.10:80
  noProxy: .example.org,.cluster.local.

So I am wondering why flux uses cluster.local. and not cluster.local (without the dot)?

Or is there a way to add an env to Kustomization or the deployment kustomize-controller?

[hiddeco]

See:

• Istio: failed to download artifact from source-controller: 503 service unavailable  #598 (comment)
• cluster.local breaks clusters which run with another clusterDomain. #724 (comment)
• Openshift Operator Kustomization not Fount 404 #1581 (comment)
• source-controller Deployment manifest expects the default cluster domain source-controller#593 (comment)

[klml]

@hiddeco thx for your hints, trailing dots in domainnames was really new for me.

I will include .cluster.local. in my Cluster noProxy, since it is a valid domain and maybe other services will use it.

[LiLister]

Same error. My case is flux run in AKS cluster. The http_proxy_config for AKS cluster is correct. But the env for the controllers of Flux isn't. After update the env (the value for no_proxy) through edit the deployment of the controller, it works.
Thanks a lot!

[rbjorklin]

For what it's worth Flux default allow-egress NetworkPolicy blocks all incoming traffic from other namespaces than where it is deploy. See policy here.