Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Wireguard NAPI threading for high packet per second clusters #9223

Open
jrcichra opened this issue Sep 12, 2024 · 0 comments
Open

Support Wireguard NAPI threading for high packet per second clusters #9223

jrcichra opened this issue Sep 12, 2024 · 0 comments

Comments

@jrcichra
Copy link

Expected Behavior

No packets are dropped if a node receives > ~180kpps.

Current Behavior

In our environment we see dropped packets at around ~180kpps coming over the wireguard interface. A single core will be at 100% CPU usage for wireguard.

Possible Solution

Support NAPI threading by setting /sys/class/net/wireguard.cali/threaded to 1. This will spread the load across many cores.

Steps to Reproduce (for bugs)

  1. Send enough pps of about ~1400 bytes to a wireguard node running a server where the traffic goes through the pod network
  2. Observe the cpu of the node and see 50% sortirq and 50% wireguard kernel threads using a core.
  3. Observe metrics for dropped receive packets such as container_network_receive_packets_dropped_total.
  4. echo 1 > /sys/class/net/wireguard.cali/threaded to resolve the issue.

Context

Kubernetes cluster with wireguard encryption cannot sustain the packets per second and is causing all tenants on a machine to experience high packet loss.

Your Environment

  • Calico version: v3.27.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.30.1
  • Operating System and version: Debian Bookworm
  • Link to your project (optional):
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant