Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gateway_discovery) enable gateway discovery in db mode #4828

Merged
merged 14 commits into from
Nov 2, 2023
Merged
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,10 @@ Adding a new version? You'll need three changes:
- KIC now specifies its `UserAgent` when communicating with kube-apiserver
as `kong-ingress-controller/${VERSION}` where version is the version of KIC.
[#5019](https://github.com/Kong/kubernetes-ingress-controller/pull/5019)
- Allow Gateway Discovery with database backed Kong. KIC will send Kong
configurations to one of the backend pods of the service specified by the
flag `--kong-admin-svc` if Kong gateway is DB backed.
[#4828](https://github.com/Kong/kubernetes-ingress-controller/pull/4828)

[KIC Annotations reference]: https://docs.konghq.com/kubernetes-ingress-controller/latest/references/annotations/
[KIC CRDs reference]: https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/
Expand Down
14 changes: 14 additions & 0 deletions config/variants/multi-gw-postgres/gateway_admin_service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: kong-admin
namespace: kong
spec:
clusterIP: "None"
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
selector:
app: proxy-kong
ports:
- name: admin
port: 8444
targetPort: 8444
protocol: TCP
113 changes: 113 additions & 0 deletions config/variants/multi-gw-postgres/gateway_deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: proxy-kong
name: proxy-kong
namespace: kong
spec:
replicas: 2
selector:
matchLabels:
app: proxy-kong
template:
metadata:
annotations:
traffic.sidecar.istio.io/includeInboundPorts: ""
kuma.io/gateway: enabled
kuma.io/service-account-token-volume: kong-serviceaccount-token
labels:
app: proxy-kong
spec:
serviceAccountName: kong-serviceaccount
automountServiceAccountToken: false
volumes:
- name: kong-serviceaccount-token
projected:
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
name: kube-root-ca.crt
items:
- key: ca.crt
path: ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
containers:
- name: proxy
image: kong-placeholder:placeholder # This is replaced by the config/image.yaml component
env:
# servers
- name: KONG_PROXY_LISTEN
value: 0.0.0.0:8000 reuseport backlog=16384, 0.0.0.0:8443 http2 ssl reuseport backlog=16384
- name: KONG_PORT_MAPS
value: "80:8000, 443:8443"
- name: KONG_ADMIN_LISTEN
value: 0.0.0.0:8444 http2 ssl reuseport backlog=16384
- name: KONG_STATUS_LISTEN
value: 0.0.0.0:8100
# DB
- name: KONG_DATABASE
value: "postgres"
- name: KONG_PG_HOST
value: postgres
- name: KONG_PG_PASSWORD
value: kong
# runtime tweaks
- name: KONG_NGINX_WORKER_PROCESSES
value: "2"
- name: KONG_KIC
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
value: "on"
# logging
- name: KONG_ADMIN_ACCESS_LOG
value: /dev/stdout
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
# - name: KONG_PROXY_ACCESS_LOG
# - value: /dev/stdout
- name: KONG_PROXY_ERROR_LOG
value: /dev/stderr
# router mode in 3.0.0. use `traditional` here for full compatibility.
- name: KONG_ROUTER_FLAVOR
value: traditional
lifecycle:
preStop:
exec:
command: [ "/bin/bash", "-c", "kong quit" ]
ports:
- name: proxy
containerPort: 8000
protocol: TCP
- name: proxy-ssl
containerPort: 8443
protocol: TCP
- name: metrics
containerPort: 8100
protocol: TCP
livenessProbe:
httpGet:
path: /status
port: 8100
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /status/ready
port: 8100
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Service
metadata:
name: kong-proxy
namespace: kong
spec:
selector:
app: proxy-kong
22 changes: 22 additions & 0 deletions config/variants/multi-gw-postgres/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: kong

resources:
- ../postgres
- gateway_admin_service.yaml
- gateway_deployment.yaml

components:
- ../../image/oss

patches:
- path: manager_multi_gateway_patch.yaml
- path: gateway_service_patch.yaml
- target:
group: apps
version: v1
kind: Deployment
name: ingress-kong
path: ./remove_proxy_container.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ingress-kong
name: ingress-kong
namespace: kong
spec:
template:
spec:
containers:
- name: ingress-controller
env:
- name: CONTROLLER_KONG_ADMIN_SVC
value: kong/kong-admin
- name: CONTROLLER_KONG_ADMIN_URL
$patch: delete
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
- op: remove
path: "/spec/template/spec/containers/1"
18 changes: 18 additions & 0 deletions deploy/single/all-in-one-postgres-multiple-gateways.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 28 additions & 0 deletions internal/clients/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/kong/kubernetes-ingress-controller/v3/internal/adminapi"
"github.com/kong/kubernetes-ingress-controller/v3/internal/util"
"github.com/kong/kubernetes-ingress-controller/v3/internal/util/clock"
dataplaneutil "github.com/kong/kubernetes-ingress-controller/v3/internal/util/dataplane"
)

// DefaultReadinessReconciliationInterval is the interval at which the manager will run readiness reconciliation loop.
Expand All @@ -29,6 +30,7 @@ type ClientFactory interface {
type AdminAPIClientsProvider interface {
KonnectClient() *adminapi.KonnectClient
GatewayClients() []*adminapi.Client
GatewayClientsToConfigure() []*adminapi.Client
}

// Ticker is an interface that allows to control a ticker.
Expand All @@ -48,6 +50,8 @@ type AdminAPIClientsManager struct {
discoveredAdminAPIsNotifyChan chan []adminapi.DiscoveredAdminAPI
gatewayClientsChangesSubscribers []chan struct{}

dbMode string
randmonkey marked this conversation as resolved.
Show resolved Hide resolved

ctx context.Context
onceNotifyLoopRunning sync.Once
runningChan chan struct{}
Expand Down Expand Up @@ -85,6 +89,13 @@ func WithReadinessReconciliationTicker(ticker Ticker) AdminAPIClientsManagerOpti
}
}

// WithDBMode allows to set the DBMode of the Kong gateway instances behind the admin API service.

pmalek marked this conversation as resolved.
Show resolved Hide resolved
func (c *AdminAPIClientsManager) WithDBMode(dbMode string) *AdminAPIClientsManager {
c.dbMode = dbMode
return c
}

func NewAdminAPIClientsManager(
ctx context.Context,
logger logr.Logger,
Expand Down Expand Up @@ -173,6 +184,23 @@ func (c *AdminAPIClientsManager) GatewayClients() []*adminapi.Client {
return lo.Values(c.readyGatewayClients)
}

func (c *AdminAPIClientsManager) GatewayClientsToConfigure() []*adminapi.Client {
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
c.lock.RLock()
defer c.lock.RUnlock()
readyGatewayClients := lo.Values(c.readyGatewayClients)
// With dbless mode, we should send configuration to ALL Kong gateway instances.
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
if dataplaneutil.IsDBLessMode(c.dbMode) {
return readyGatewayClients
}
// When Kong gateway is DB backed, we return a random admin API client
// since KIC only need to send requests to one instance.
// If there are no ready gateway clients, we return an empty list.
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
if len(readyGatewayClients) == 0 {
return []*adminapi.Client{}
}
return readyGatewayClients[:1]
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
}

func (c *AdminAPIClientsManager) GatewayClientsCount() int {
c.lock.RLock()
defer c.lock.RUnlock()
Expand Down
47 changes: 45 additions & 2 deletions internal/clients/manager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,12 @@ func TestAdminAPIClientsManager_OnNotifyClientsAreUpdatedAccordingly(t *testing.
}

func TestNewAdminAPIClientsManager_NoInitialClientsDisallowed(t *testing.T) {
_, err := clients.NewAdminAPIClientsManager(context.Background(), zapr.NewLogger(zap.NewNop()), nil, &mockReadinessChecker{})
_, err := clients.NewAdminAPIClientsManager(
context.Background(),
zapr.NewLogger(zap.NewNop()),
nil,
&mockReadinessChecker{},
)
require.ErrorContains(t, err, "at least one initial client must be provided")
}

Expand Down Expand Up @@ -196,6 +201,7 @@ func TestAdminAPIClientsManager_Clients(t *testing.T) {
require.NoError(t, err)
require.Len(t, m.GatewayClients(), 1, "expecting one initial client")
require.Equal(t, m.GatewayClientsCount(), 1, "expecting one initial client")
require.Len(t, m.GatewayClientsToConfigure(), 1, "Expecting one initial client")

konnectTestClient := &adminapi.KonnectClient{}
m.SetKonnectClient(konnectTestClient)
Expand All @@ -204,6 +210,36 @@ func TestAdminAPIClientsManager_Clients(t *testing.T) {
require.Equal(t, konnectTestClient, m.KonnectClient(), "konnect client should be returned from KonnectClient")
}

func TestAdminAPIClientsManager_Clients_DBMode(t *testing.T) {
testClient, err := adminapi.NewTestClient("localhost:8080")
require.NoError(t, err)
testClient2, err := adminapi.NewTestClient("localhost:8081")
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
require.NoError(t, err)
initialClients := []*adminapi.Client{testClient, testClient2}
require.NoError(t, err)

m, err := clients.NewAdminAPIClientsManager(
context.Background(),
zapr.NewLogger(zap.NewNop()),
initialClients,

randmonkey marked this conversation as resolved.
Show resolved Hide resolved
&mockReadinessChecker{},
)
require.NoError(t, err)
m = m.WithDBMode("postgres")

clients := m.GatewayClients()
require.Len(t, clients, 2, "Expecting 2 clients returned with DB mode")

configureClients := m.GatewayClientsToConfigure()
require.Len(t, configureClients, 1, "Expecting 1 client to configure")
require.Truef(t, lo.ContainsBy(initialClients, func(c *adminapi.Client) bool {
return c.BaseRootURL() == configureClients[0].BaseRootURL()
}), "Client's address %s should be in initial clients")

require.Equal(t, m.GatewayClientsCount(), 2, "Expecting 2 initial clients")
}

func TestAdminAPIClientsManager_SubscribeToGatewayClientsChanges(t *testing.T) {
t.Parallel()

Expand All @@ -212,7 +248,13 @@ func TestAdminAPIClientsManager_SubscribeToGatewayClientsChanges(t *testing.T) {
require.NoError(t, err)

ctx, cancel := context.WithCancel(context.Background())
m, err := clients.NewAdminAPIClientsManager(ctx, zapr.NewLogger(zap.NewNop()), []*adminapi.Client{testClient}, readinessChecker)
m, err := clients.NewAdminAPIClientsManager(
ctx,
zapr.NewLogger(zap.NewNop()),
[]*adminapi.Client{testClient},

randmonkey marked this conversation as resolved.
Show resolved Hide resolved
readinessChecker)

require.NoError(t, err)

t.Run("no notify loop running should return false when subscribing", func(t *testing.T) {
Expand Down Expand Up @@ -430,6 +472,7 @@ func TestAdminAPIClientsManager_PeriodicReadinessReconciliation(t *testing.T) {
ctx,
zapr.NewLogger(zap.NewNop()),
[]*adminapi.Client{testClient},

randmonkey marked this conversation as resolved.
Show resolved Hide resolved
readinessChecker,
clients.WithReadinessReconciliationTicker(readinessTicker),
)
Expand Down
26 changes: 23 additions & 3 deletions internal/dataplane/kong_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -454,15 +454,35 @@ func (c *KongClient) sendOutToGatewayClients(
ctx context.Context, s *kongstate.KongState, config sendconfig.Config,
) ([]string, error) {
gatewayClients := c.clientsProvider.GatewayClients()
c.logger.V(util.DebugLevel).Info("sending configuration to gateway clients", "count", len(gatewayClients))
shas, err := iter.MapErr(gatewayClients, func(client **adminapi.Client) (string, error) {
if len(gatewayClients) == 0 {
c.logger.Error(
errors.New("no ready gateway clients"),
"Could not send configuration to gateways",
)
// Should not store the configuration in last valid config because the configuration is not validated on Kong gateway.
return c.SHAs, nil
}

gatewayClientsToConfigure := c.clientsProvider.GatewayClientsToConfigure()
configureGatewayClientURLs := lo.Map(gatewayClientsToConfigure, func(cl *adminapi.Client, _ int) string { return cl.BaseRootURL() })
c.logger.V(util.DebugLevel).Info("Sending configuration to gateway clients", "urls", configureGatewayClientURLs)

shas, err := iter.MapErr(gatewayClientsToConfigure, func(client **adminapi.Client) (string, error) {
return c.sendToClient(ctx, *client, s, config)
})
if err != nil {
return nil, err
}
previousSHAs := c.SHAs

// After succeeded to update configurations in DB mode, we should propagate the SHA from
// the chosen gateway client to other clients.
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
if !dataplaneutil.IsDBLessMode(c.dbmode) && len(shas) > 0 {
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
for _, client := range gatewayClients {
client.SetLastConfigSHA([]byte(shas[0]))
randmonkey marked this conversation as resolved.
Show resolved Hide resolved
}
}

previousSHAs := c.SHAs
sort.Strings(shas)
c.SHAs = shas

Expand Down
Loading