Can Curl out of a pod on Http but not Https - fresh install on Ubuntu 22.04 #5427
Labels
bug
Something isn't working
Comments
|
Oh wow, this is weird
So the connection itself can be made, so the traffic does go somewhere.
Sounds something is messing up the traffic and thus the SSL handshake fails.
Do you see similar SSL level errors on LE pods? Which CNI are you using? I'll throw couple things to test just to rule out stuff:
|
|
Trying with netshoot on fresh 1.31.3 cluster on Ubuntu 22.04, I see this: But if I force the curl command to use port 80 with https I get the exact same issue: So I wonder what could happen in your network so that google port 443 would get translated into port 80? What happens in your env if you force port 443 for plain HTTP? E.g. with: |
|
Hi, thanks for coming back! Its verrry strange... I'll try and add as much info as I can here...
CNI: Explicit Port: shots fired IpTables dump: So, it would appear from the curl test above that port 80 is being mapped to 443 ??? |
|
oh, also no proxy btw.. |
Not necessarily a port map... I suspect that a whole other server is responding. The curl output says "404 page not found" which is definitely not the expected response from the Google webserver. So you're connecting to something else. |
|
So this is a traceroute, I also checked the ip address with a reverseip lookup and its google! https://mxtoolbox.com/SuperTool.aspx?action=ptr%3a172.217.16.228&run=toolpage# Also ran the curl explicitly to port 433 cmd on the host, different results but equally no problems with https there either.. |
|
The IP address |
|
The open ports on it: |
|
|
|
Could you try with that traces the "real" connection to the requested port. As you see very different[1] responses from us (myself & @twz123 ), we think there's something in your network path that routes the SSL calls to some other service for some reason. Maybe because the traffic originates from the pod network with [1] This is what I see: Notice that I see If I connect directly to the IP on https: Notice the special server cert the server sends me 😄 |
|
hi, had a chance to run the traceroute... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Before creating an issue, make sure you've checked the following:
Platform
uname -a Linux tom-nuc 6.8.0-51-generic #52~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Dec 9 15:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/LinuxVersion
v1.31.3+k0s.0
Sysinfo
`k0s sysinfo`
Total memory: 15.5 GiB (pass) File system of /var/lib/k0s: ext4 (pass) Disk space available for /var/lib/k0s: 388.7 GiB (pass) Relative disk space available for /var/lib/k0s: 85% (pass) Name resolution: localhost: [127.0.0.1] (pass) Operating system: Linux (pass) Linux kernel release: 6.8.0-51-generic (pass) Max. file descriptors per process: current: 1048576 / max: 1048576 (pass) AppArmor: active (pass) Executable in PATH: modprobe: /usr/sbin/modprobe (pass) Executable in PATH: mount: /usr/bin/mount (pass) Executable in PATH: umount: /usr/bin/umount (pass) /proc file system: mounted (0x9fa0) (pass) Control Groups: version 2 (pass) cgroup controller "cpu": available (is a listed root controller) (pass) cgroup controller "cpuacct": available (via cpu in version 2) (pass) cgroup controller "cpuset": available (is a listed root controller) (pass) cgroup controller "memory": available (is a listed root controller) (pass) cgroup controller "devices": available (device filters attachable) (pass) cgroup controller "freezer": available (cgroup.freeze exists) (pass) cgroup controller "pids": available (is a listed root controller) (pass) cgroup controller "hugetlb": available (is a listed root controller) (pass) cgroup controller "blkio": available (via io in version 2) (pass) CONFIG_CGROUPS: Control Group support: built-in (pass) CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass) CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass) CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass) CONFIG_CPUSETS: Cpuset support: built-in (pass) CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass) CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass) CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass) CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass) CONFIG_BLK_CGROUP: Block IO controller: built-in (pass) CONFIG_NAMESPACES: Namespaces support: built-in (pass) CONFIG_UTS_NS: UTS namespace: built-in (pass) CONFIG_IPC_NS: IPC namespace: built-in (pass) CONFIG_PID_NS: PID namespace: built-in (pass) CONFIG_NET_NS: Network namespace: built-in (pass) CONFIG_NET: Networking support: built-in (pass) CONFIG_INET: TCP/IP networking: built-in (pass) CONFIG_IPV6: The IPv6 protocol: built-in (pass) CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass) CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass) CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass) CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass) CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass) CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass) CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass) CONFIG_NETFILTER_XT_SET: set target and match support: module (pass) CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass) CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass) CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass) CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass) CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass) CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass) CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass) CONFIG_NETFILTER_NETLINK: module (pass) CONFIG_NF_NAT: module (pass) CONFIG_IP_SET: IP set support: module (pass) CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass) CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass) CONFIG_IP_VS: IP virtual server support: module (pass) CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass) CONFIG_IP_VS_SH: Source hashing scheduling: module (pass) CONFIG_IP_VS_RR: Round-robin scheduling: module (pass) CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass) CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass) CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning) CONFIG_IP_NF_IPTABLES: IP tables support: module (pass) CONFIG_IP_NF_FILTER: Packet filtering: module (pass) CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass) CONFIG_IP_NF_NAT: iptables NAT support: module (pass) CONFIG_IP_NF_MANGLE: Packet mangling: module (pass) CONFIG_NF_DEFRAG_IPV4: module (pass) CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning) CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass) CONFIG_IP6_NF_FILTER: Packet filtering: module (pass) CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass) CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass) CONFIG_NF_DEFRAG_IPV6: module (pass) CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass) CONFIG_LLC: module (pass) CONFIG_STP: module (pass) CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass) CONFIG_PROC_FS: /proc file system support: built-in (pass)What happened?
Can make http calls outbound to the internet but not https. This is on a blank install running a pretty standard, up to date test container. I have seen this error elsewhere in other containers so it appears a generic problem.
Steps to reproduce
kubectl run -i --tty --rm debug --image=nicolaka/netshoot --restart=Never -- bashWithin the shell that comes up I can run this ok:
curl -v http://www.google.combut when I run it for https:
curl -v httpS://www.google.comcurl -v https://www.google.com
curl: (35) OpenSSL/3.3.0: error:0A0000C6:SSL routines::packet length too long
My DNS resolves ok within the container and on the host - this is a single node - my firewall (UFW) is disabled.
Expected behavior
Obviously, I expect to be able to connect to the internet using SSL. 👍
(This is messing with Lets Encrypt initialisation which is why I'm really here..)
Actual behavior
I get a random cryptographic error. The same curl command works perfectly on the host - ironically on an earlier version of curl.
Container curl info:
Host curl info:
Screenshots and logs
No response
Additional context
Happy to provide extra logs, info as required. Finding K0s to otherwise be awesome!
The text was updated successfully, but these errors were encountered: