Ingress won't find internal service/pod - curl: (52) Empty reply from server

[emeraldhieu]

Hello, I want to send the below request to the pod from the public via k3d's loadbalancer. I have followed this guide https://k3d.io/v5.3.0/usage/exposing_services but my request responded "curl: (52) Empty reply from server".

Request

curl --location 'http://localhost:8080/orders' \
--header 'Content-Type: application/json' \
--data '{
    "products": [
        "coke",
        "juice",
        "cider"
    ]
}'

Response

curl: (52) Empty reply from server

K3d's version

[~]$ k3d --version
k3d version v5.4.9
k3s version v1.25.7-k3s1 (default)

K3d's containers

[~]$ docker container ls                  
CONTAINER ID   IMAGE                      COMMAND                  CREATED        STATUS        PORTS                                                                                             NAMES
b301ff794577   55aa42e8234e               "/bin/sh -c nginx-pr…"   15 hours ago   Up 14 hours   0.0.0.0:9900->80/tcp, 0.0.0.0:55303->6443/tcp, 0.0.0.0:8080->50001/tcp, 0.0.0.0:9901->50001/tcp   k3d-cluster42-serverlb
3aa95c426a45   rancher/k3s:v1.25.7-k3s1   "/bin/k3d-entrypoint…"   2 days ago     Up 14 hours                                                                                                     k3d-cluster42-server-0
4b3e88567a7e   registry:2                 "/entrypoint.sh /etc…"   2 days ago     Up 14 hours   0.0.0.0:5050->5000/tcp                                                                            k3d-registry42

Logs from k3d's loadbalancer

[~]$ docker logs -f k3d-cluster42-serverlb
...
===== Initial nginx configuration =====
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
###################################
# Generated by confd 2023-04-17 19:26:19.620660216 +0000 UTC m=+0.013200011 #
#             #######             #
#             # k3d #             #
#             #######             #
###################################

error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1024;
}

stream {

  upstream 50001_tcp {
    server k3d-cluster42-server-0:50001 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        50001;
    proxy_pass    50001_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

  upstream 6443_tcp {
    server k3d-cluster42-server-0:6443 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        6443;
    proxy_pass    6443_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

  upstream 80_tcp {
    server k3d-cluster42-server-0:80 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        80;
    proxy_pass    80_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

}

=======================================
2023/04/17 19:26:21 [notice] 34#34: using the "epoll" event method
2023/04/17 19:26:21 [notice] 34#34: nginx/1.19.10
2023/04/17 19:26:21 [notice] 34#34: built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1) 
2023/04/17 19:26:21 [notice] 34#34: OS: Linux 5.15.49-linuxkit
2023/04/17 19:26:21 [notice] 34#34: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/04/17 19:26:21 [notice] 34#34: start worker processes
2023/04/17 19:26:21 [notice] 34#34: start worker process 39
2023/04/17 19:26:21 [notice] 34#34: start worker process 40
2023/04/17 19:26:21 [notice] 34#34: start worker process 41
2023/04/17 19:26:21 [notice] 34#34: start worker process 42
2023/04/17 19:26:21 [notice] 34#34: start worker process 43
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO Backend set to file
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO Starting confd
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO Backend source(s) set to /etc/confd/values.yaml
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Loading template resources from confdir /etc/confd
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Found template: /etc/confd/conf.d/nginx.toml
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Loading template resource from /etc/confd/conf.d/nginx.toml
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Retrieving keys from store
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Key prefix set to /
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Key Map: map[string]string{"/ports/50001.tcp/0":"k3d-cluster42-server-0", "/ports/6443.tcp/0":"k3d-cluster42-server-0", "/ports/80.tcp/0":"k3d-cluster42-server-0", "/settings/workerConnections":"1024"}
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Got the following map from store: map[/ports/50001.tcp/0:k3d-cluster42-server-0 /ports/6443.tcp/0:k3d-cluster42-server-0 /ports/80.tcp/0:k3d-cluster42-server-0 /settings/workerConnections:1024]
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Using source template /etc/confd/templates/nginx.tmpl
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Compiling source template /etc/confd/templates/nginx.tmpl
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Comparing candidate config to /etc/nginx/nginx.conf
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO /etc/nginx/nginx.conf has md5sum 4d0edc22e87dda895f19262c57ab3b9a should be 0d48be4af59049466a72faa22b36493a
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO Target config /etc/nginx/nginx.conf out of sync
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Running /usr/sbin/nginx -T -c /etc/nginx/.nginx.conf587714234
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG "nginx: the configuration file /etc/nginx/.nginx.conf587714234 syntax is ok\nnginx: configuration file /etc/nginx/.nginx.conf587714234 test is successful\n# configuration file /etc/nginx/.nginx.conf587714234:\n###################################\n# Generated by confd 2023-04-17 19:26:21.682044927 +0000 UTC m=+0.013428553 #\n#             #######             #\n#             # k3d #             #\n#             #######             #\n###################################\n\nerror_log stderr notice;\n\nworker_processes auto;\nevents {\n  multi_accept on;\n  use epoll;\n  worker_connections 1024;\n}\n\nstream {\n\n  upstream 50001_tcp {\n    server k3d-cluster42-server-0:50001 max_fails=1 fail_timeout=10s;\n  }\n\n  server {\n    listen        50001;\n    proxy_pass    50001_tcp;\n    proxy_timeout 600;\n    proxy_connect_timeout 2s;\n  }\n\n  upstream 6443_tcp {\n    server k3d-cluster42-server-0:6443 max_fails=1 fail_timeout=10s;\n  }\n\n  server {\n    listen        6443;\n    proxy_pass    6443_tcp;\n    proxy_timeout 600;\n    proxy_connect_timeout 2s;\n  }\n\n  upstream 80_tcp {\n    server k3d-cluster42-server-0:80 max_fails=1 fail_timeout=10s;\n  }\n\n  server {\n    listen        80;\n    proxy_pass    80_tcp;\n    proxy_timeout 600;\n    proxy_connect_timeout 2s;\n  }\n\n}\n\n"
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Overwriting target config /etc/nginx/nginx.conf
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG Running /usr/sbin/nginx -s reload
2023/04/17 19:26:21 [notice] 34#34: signal 1 (SIGHUP) received from 48, reconfiguring
2023/04/17 19:26:21 [notice] 34#34: reconfiguring
2023/04/17 19:26:21 [notice] 34#34: using the "epoll" event method
2023/04/17 19:26:21 [notice] 34#34: start worker processes
2023/04/17 19:26:21 [notice] 34#34: start worker process 49
2023/04/17 19:26:21 [notice] 34#34: start worker process 50
2023/04/17 19:26:21 [notice] 34#34: start worker process 51
2023/04/17 19:26:21 [notice] 34#34: start worker process 52
2023/04/17 19:26:21 [notice] 34#34: start worker process 53
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: DEBUG "2023/04/17 19:26:21 [notice] 48#48: signal process started\n"
2023-04-17T19:26:21Z k3d-cluster42-serverlb confd[33]: INFO Target config /etc/nginx/nginx.conf has been updated
2023/04/17 19:26:21 [notice] 39#39: gracefully shutting down
2023/04/17 19:26:21 [notice] 41#41: gracefully shutting down
2023/04/17 19:26:21 [notice] 40#40: gracefully shutting down
2023/04/17 19:26:21 [notice] 41#41: exiting
2023/04/17 19:26:21 [notice] 39#39: exiting
2023/04/17 19:26:21 [notice] 40#40: exiting
2023/04/17 19:26:21 [notice] 40#40: exit
2023/04/17 19:26:21 [notice] 39#39: exit
2023/04/17 19:26:21 [notice] 41#41: exit
2023/04/17 19:26:21 [notice] 42#42: gracefully shutting down
2023/04/17 19:26:21 [notice] 43#43: gracefully shutting down
2023/04/17 19:26:21 [notice] 43#43: exiting
2023/04/17 19:26:21 [notice] 42#42: exiting
2023/04/17 19:26:21 [notice] 43#43: exit
2023/04/17 19:26:21 [notice] 42#42: exit
2023/04/17 19:26:21 [notice] 34#34: signal 17 (SIGCHLD) received from 41
2023/04/17 19:26:21 [notice] 34#34: worker process 39 exited with code 0
2023/04/17 19:26:21 [notice] 34#34: worker process 40 exited with code 0
2023/04/17 19:26:21 [notice] 34#34: worker process 41 exited with code 0
2023/04/17 19:26:21 [notice] 34#34: worker process 42 exited with code 0
2023/04/17 19:26:21 [notice] 34#34: signal 29 (SIGIO) received
2023/04/17 19:26:21 [notice] 34#34: signal 17 (SIGCHLD) received from 42
2023/04/17 19:26:21 [notice] 34#34: worker process 43 exited with code 0
2023/04/17 19:26:21 [notice] 34#34: signal 29 (SIGIO) received
2023/04/17 19:26:38 [error] 49#49: *11 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:50001, upstream: "172.20.0.3:50001", bytes from/to client:0/0, bytes from/to upstream:0/0
2023/04/17 19:26:41 [error] 49#49: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:50001, upstream: "172.20.0.3:50001", bytes from/to client:0/0, bytes from/to upstream:0/0
2023/04/17 20:03:12 [error] 49#49: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:50001, upstream: "172.20.0.3:50001", bytes from/to client:0/0, bytes from/to upstream:0/0
2023/04/18 08:42:56 [error] 49#49: *43 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:50001, upstream: "172.20.0.3:50001", bytes from/to client:0/0, bytes from/to upstream:0/0

Service

[~]$ kubectl get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
kubernetes        ClusterIP   10.43.0.1       <none>        443/TCP                          2d14h
order             NodePort    10.43.29.182    <none>        50001:32353/TCP                  80m

Service described

[~]$ kubectl describe svc order
Name:                     order
Namespace:                default
Labels:                   app=order
Annotations:              <none>
Selector:                 app=order
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.29.182
IPs:                      10.43.29.182
Port:                     <unset>  50001/TCP
TargetPort:               50001/TCP
NodePort:                 <unset>  32353/TCP
Endpoints:                10.42.0.60:50001
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Pod

[~]$ kubectl get pod
NAME                               READY   STATUS    RESTARTS      AGE
order-5ff786d595-sxbmr             1/1     Running   0             81m

Pod described

[~]$ kubectl describe pod order-5ff786d595-sxbmr
Name:             order-5ff786d595-sxbmr
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3d-cluster42-server-0/172.20.0.3
Start Time:       Tue, 18 Apr 2023 15:42:22 +0700
Labels:           app=order
                  pod-template-hash=5ff786d595
Annotations:      <none>
Status:           Running
IP:               10.42.0.60
IPs:
  IP:           10.42.0.60
Controlled By:  ReplicaSet/order-5ff786d595
Containers:
  order:
    Container ID:   containerd://9335d461c1ee4fda9b1a29bc7895b34e4b17db877f790e449eadf4f89ef5d0b8
    Image:          k3d-registry42:5050/order:1.0-SNAPSHOT
    Image ID:       k3d-registry42:5050/order@sha256:7aca095a64ff8adb9e15f38d810ba3f965d8693fd902bffb970b1360c2128eee
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 18 Apr 2023 15:42:28 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  256Mi
    Environment Variables from:
      order-cm    ConfigMap  Optional: false
      order-s     Secret     Optional: false
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-64crw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-64crw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Ingress

[~]$ kubectl get ing    
NAME          CLASS     HOSTS   ADDRESS      PORTS   AGE
ingress-ing   traefik   *       172.20.0.3   80      82m

Ingress described

[~]$ ku describe ing ingress-ing
Name:             ingress-ing
Labels:           <none>
Namespace:        default
Address:          172.20.0.3
Ingress Class:    traefik
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   order:50001 (10.42.0.60:50001)
Annotations:  ingress.kubernetes.io/ssl-redirect: false
Events:       <none>

Ingress manifest

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-ing
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: order
                port:
                  number: 50001

---

"172.20.0.3:50001" is the ingress. Why won't the ingress find the pod while it already routes to service "order:50001". What am I missing or misunderstading here?

[emeraldhieu]

I've fixed it by changing service's type to LoadBalancer and adjusting containerPort(s).