Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determine if network policies are applied #298

Open
pelzerim opened this issue Aug 14, 2024 · 1 comment
Open

Determine if network policies are applied #298

pelzerim opened this issue Aug 14, 2024 · 1 comment
Labels
needs investigation question Further information is requested

Comments

@pelzerim
Copy link

What happened:

We have workloads that require individual pods per run. This results in a very high pod churn and pods come and go at a high rate.

The network policies can in some cases not be applied when the workload runs and the delayed application causes canceled network requests. These are not acceptable in our setup.

As a workaround we tried strict mode but have determined it to be too unstable (25% failure rate after a while to setup netpols).

The POD_IP_ANNOTATION do speed up the process but are still not enough to ensure a 0% failure rate.

We now run an init container with every pod that ensures that the network policies are setup correctly. It literally watches for Successfully attached.*${POD_NAME} inside /opt/k8s/network-policy-logs.log. This however still does not work reliably.

Is there any way to determine of the network policies are applied correctly? Can we determine this somehow from the low level information here?

Environment:

  • Kubernetes version (use kubectl version): 1.30
  • CNI Version: v1.18.2-eksbuild.1
  • Network Policy Agent Version: aws-network-policy-agent:v1.1.2-eksbuild.1
  • OS (e.g: cat /etc/os-release):
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
  • Kernel (e.g. uname -a):
Linux ip-10-0-86-101.eu-west-2.compute.internal 5.10.217-205.860.amzn2.aarch64 #1 SMP Tue May 21 16:52:27 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
@pelzerim pelzerim added needs investigation question Further information is requested labels Aug 14, 2024
@Pavani-Panakanti
Copy link
Contributor

@pelzerim We fixed a race condition in latest release with strict mode where network policies might not be applied in high pod churn cases #306. Can you try using latest image https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.18.5 and let us know if it fixes the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs investigation question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants