Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discuss: work around SecurityRuleAddressesOrPortsPerSecurityGroupLimitReached when service floating ip disabled #5919

Closed
ArchangelSDY opened this issue Apr 15, 2024 · 8 comments · May be fixed by #7190
Closed

Discuss: work around SecurityRuleAddressesOrPortsPerSecurityGroupLimitReached when service floating ip disabled #5919

ArchangelSDY opened this issue Apr 15, 2024 · 8 comments · May be fixed by #7190
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@ArchangelSDY
Copy link
Contributor

I'd like to discuss potential solutions for issue #2725 .

The background is we have a large cluster with > 500 nodes and > 30 services. These services have floating ip disabled, so in a NSG rule all backend node IPs are listed as dest addresses. However, NSG has a limit that sum of IPs among all rules must not exceed 4000. In above cluster the value is 500 * 30 = 15000, exceeding the limit a lot.

I don't think NSG has plan to increase such limit. So the only way in my mind is that maybe we can provide an annotation to let user explicitly specify NSG dest addresses(probably a CIDR list), instead of using all backend node IPs.

For example:

"service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip-nsg-dest": "10.1.0.0/16,10.2.0.0/16"

Then we create a NSG rule with destinationAddressPrefix set to 10.1.0.0/16,10.2.0.0/16. It would only count as 2 to the limit.

Thoughts? I'm happy to submit a PR if you're ok with the change.
@ArchangelSDY ArchangelSDY added the kind/feature Categorizes issue or PR as related to a new feature. label Apr 15, 2024
@zarvd
Copy link
Contributor

zarvd commented Jun 21, 2024

I am thinking maybe we could set the backend node subnet CIDR as the default destination addresses by default, without requiring additional annotations or settings from the user. Need to double check if it’s feasible.

@nilo19
Copy link
Contributor

nilo19 commented Jun 24, 2024

I am thinking maybe we could set the backend node subnet CIDR as the default destination addresses by default, without requiring additional annotations or settings from the user. Need to double check if it’s feasible.

Will we lose security if open up all subnet while most of the space may not be used?

@nilo19
Copy link
Contributor

nilo19 commented Jun 24, 2024

For local services, maybe we can only open nodes where there are endpoints running.

@zarvd
Copy link
Contributor

zarvd commented Jun 24, 2024

Will we lose security if open up all subnet while most of the space may not be used?

Yes, unfortunately, we would. Had a thought that set CIDR iff nodes * services exceeds the limit; otherwise, set node IP list. Still not sure about the idea.

@zarvd zarvd mentioned this issue Sep 9, 2024
Merged
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 22, 2024
@zarvd zarvd mentioned this issue Oct 2, 2024
Open
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 22, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 21, 2024
@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Aggregate NSG destination addresses optimally zarvd/cloud-provider-azure
5 participants
@ArchangelSDY @zarvd @k8s-ci-robot @nilo19 @k8s-triage-robot