Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gateway_discovery) enable gateway discovery in db mode #4828

Merged
merged 14 commits into from
Nov 2, 2023

Conversation

randmonkey
Copy link
Contributor

@randmonkey randmonkey commented Oct 12, 2023

What this PR does / why we need it:

enables gateway discovery in DB mode. In DB mode, if multiple gateways are behind the Kong admin API service, KIC will discover all ready clients and send config to one of them in sending configurations.

Which issue this PR fixes:

fixes #4751

Special notes for your reviewer:

This PR could not cover the case where multiple gateways are using different DBs. This case is tracked in #4845.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

  • the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

@randmonkey randmonkey requested a review from a team as a code owner October 12, 2023 09:17
@randmonkey randmonkey marked this pull request as draft October 12, 2023 09:17
@randmonkey randmonkey self-assigned this Oct 12, 2023
@randmonkey randmonkey added the area/feature New feature or request label Oct 12, 2023
@codecov
Copy link

codecov bot commented Oct 12, 2023

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (c44cb7b) 75.5% compared to head (56a71a4) 75.5%.
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #4828     +/-   ##
=======================================
- Coverage   75.5%   75.5%   -0.1%     
=======================================
  Files        167     167             
  Lines      18881   18903     +22     
=======================================
+ Hits       14273   14278      +5     
- Misses      3783    3795     +12     
- Partials     825     830      +5     
Files Coverage Δ
internal/manager/config_validation.go 82.1% <ø> (-1.6%) ⬇️
internal/manager/run.go 55.5% <100.0%> (+0.6%) ⬆️
internal/util/dataplane/mode.go 100.0% <100.0%> (ø)
internal/clients/manager.go 96.2% <80.0%> (-1.5%) ⬇️
internal/dataplane/kong_client.go 86.0% <78.9%> (-3.5%) ⬇️

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@randmonkey randmonkey added the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6503776252

@team-k8s-bot team-k8s-bot removed the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
@randmonkey randmonkey force-pushed the feat/send_to_one_gw_db_backed branch 2 times, most recently from 06417b3 to 7d93499 Compare October 13, 2023 07:15
@randmonkey randmonkey changed the title [WIP] enable gateway discovery in db mode feat(gateway_discovery) enable gateway discovery in db mode Oct 13, 2023
@randmonkey randmonkey added the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6505244182

@team-k8s-bot team-k8s-bot removed the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
@randmonkey randmonkey marked this pull request as ready for review October 13, 2023 07:24
@randmonkey randmonkey added the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6505946279

@team-k8s-bot team-k8s-bot removed the ci/run-e2e Trigger e2e test run from PR label Oct 13, 2023
Copy link
Contributor

@rainest rainest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to clarify what exactly of discovery we wanted to port. Some aspects of it don't really make sense for DB mode.

I think what we actually want to target is that both DB and DB-less mode only require indicating the admin Service in configuration, but not change how DB mode sends config.

We have the configuring the target admin service as a prerequisite for discovery already, so we should simply send a single batch of update requests to the Service, and let kube-proxy deal with distributing that to a ready instance per its normal rules. We don't need to send updates to every replica, so we shouldn't need to bypass kube-proxy and write our own instance logic.

Use of ClusterIP=None a in DB-less discovery though, and I don't know if there's a good way to reconcile that without requiring different Service config for DB mode. We could send a single request to the headless Service hostname and let DNS select an instance at random.

Using our own logic will ignore broader system networking policy. It wouldn't honor something like https://kuma.io/docs/2.4.x/policies/locality-aware/, for example. I don't know how well headless plays with that in general, so that's maybe wrapped up in #4698.

We presumably do want to handle mTLS, but it looks this is handled somewhat outside discovery itself--the cert configuration takes effect regardless of whether the request comes through kube-proxy or not AFAICT.

@randmonkey randmonkey force-pushed the feat/send_to_one_gw_db_backed branch 4 times, most recently from 11e674c to a8b9162 Compare October 24, 2023 08:13
@randmonkey randmonkey added the ci/run-e2e Trigger e2e test run from PR label Oct 24, 2023
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6624257331

internal/clients/manager_test.go Outdated Show resolved Hide resolved
internal/clients/manager_test.go Outdated Show resolved Hide resolved
internal/clients/manager_test.go Outdated Show resolved Hide resolved
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
internal/clients/manager.go Outdated Show resolved Hide resolved
internal/clients/manager.go Outdated Show resolved Hide resolved
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6731367753

@team-k8s-bot team-k8s-bot removed the ci/run-e2e Trigger e2e test run from PR label Nov 2, 2023
internal/clients/manager.go Show resolved Hide resolved
internal/dataplane/kong_client.go Show resolved Hide resolved
internal/dataplane/kong_client.go Outdated Show resolved Hide resolved
@randmonkey randmonkey added the ci/run-e2e Trigger e2e test run from PR label Nov 2, 2023
@team-k8s-bot
Copy link
Collaborator

E2E (targeted) tests with KIND-based clusters were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6734361214

@team-k8s-bot team-k8s-bot removed the ci/run-e2e Trigger e2e test run from PR label Nov 2, 2023
internal/clients/manager.go Outdated Show resolved Hide resolved
czeslavo
czeslavo previously approved these changes Nov 2, 2023
Copy link
Contributor

@czeslavo czeslavo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

internal/clients/manager.go Outdated Show resolved Hide resolved
internal/clients/manager.go Outdated Show resolved Hide resolved
@randmonkey randmonkey dismissed rainest’s stale review November 2, 2023 15:46

Fixed the SHA tracking problem.

@pmalek
Copy link
Member

pmalek commented Nov 2, 2023

@rainest Do you have any comments still that you'd like to raise here?

Copy link
Contributor

@rainest rainest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer seeing extra updates after the SHA change, so I no longer have anything:

2023-11-02T16:41:32Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:32Z	info	Successfully synced configuration to Kong	{"url": "https://10.244.0.59:8444", "update_strategy": "DBMode", "v": 0}
2023-11-02T16:41:35Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:38Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:41Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:44Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:47Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}
2023-11-02T16:41:50Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.60:8444"]}
2023-11-02T16:41:53Z	debug	Sending configuration to gateway clients	{"v": 1, "urls": ["https://10.244.0.59:8444"]}

@rainest rainest merged commit c39e31e into main Nov 2, 2023
35 checks passed
@rainest rainest deleted the feat/send_to_one_gw_db_backed branch November 2, 2023 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/feature New feature or request size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Gateway Discovery for DB mode
6 participants