Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: TLS listeners supported #112

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add one more sample with just the TLSRoute and a Gateway with TLS listener?

Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@ metadata:
name: kong
spec:
controllerName: konghq.com/gateway-operator
parametersRef:
group: gateway-operator.konghq.com
kind: GatewayConfiguration
name: kong
namespace: default
---
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
Expand All @@ -24,3 +19,8 @@ spec:
- name: http2
protocol: HTTP
port: 8089
- name: tls
protocol: TLS
port: 9443
tls:
mode: Passthrough
12 changes: 12 additions & 0 deletions controller/controlplane/controller_utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,14 @@ func TestSetControlPlaneDefaults(t *testing.T) {
Name: "CONTROLLER_ADMISSION_WEBHOOK_LISTEN",
Value: consts.ControlPlaneAdmissionWebhookEnvVarValue,
},
{
Name: "",
Value: consts.ControlPlaneAdmissionWebhookEnvVarValue,
},
Comment on lines +437 to +440
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we don't need this right?

{
Name: "CONTROLLER_FEATURE_GATES",
Value: "GatewayAlpha=true",
},
},
},
},
Expand Down Expand Up @@ -511,6 +519,10 @@ func TestSetControlPlaneDefaults(t *testing.T) {
Name: "CONTROLLER_ADMISSION_WEBHOOK_LISTEN",
Value: consts.ControlPlaneAdmissionWebhookEnvVarValue,
},
{
Name: "CONTROLLER_FEATURE_GATES",
Value: "GatewayAlpha=true",
},
},
},
},
Expand Down
6 changes: 6 additions & 0 deletions controller/gateway/controller_conditions.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,10 @@ const (
// to express that more than one TLS secret has been set in the listener
// TLS configuration.
ListenerReasonTooManyTLSSecrets k8sutils.ConditionReason = "TooManyTLSSecrets"

// ListenereReasonInvalidTLSMode must be used with the Accepted condition
// to express that the listener has an invalid TLS mode.
// HTTPS can only be configured with mode Terminate, while TLS can only be
// be configured with mode Passthrough.
ListenereReasonInvalidTLSMode k8sutils.ConditionReason = "InvalidTLSMode"
Comment on lines +36 to +40
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// ListenereReasonInvalidTLSMode must be used with the Accepted condition
// to express that the listener has an invalid TLS mode.
// HTTPS can only be configured with mode Terminate, while TLS can only be
// be configured with mode Passthrough.
ListenereReasonInvalidTLSMode k8sutils.ConditionReason = "InvalidTLSMode"
// ListenerReasonInvalidTLSMode must be used with the Accepted condition
// to express that the listener has an invalid TLS mode.
// HTTPS can only be configured with mode Terminate, while TLS can only be
// be configured with mode Passthrough.
ListenerReasonInvalidTLSMode k8sutils.ConditionReason = "InvalidTLSMode"

We might add a reference link to https://gateway-api.sigs.k8s.io/guides/tls/#clientserver-and-tls which contains the table with allowed modes for particular route types.

)
128 changes: 72 additions & 56 deletions controller/gateway/controller_reconciler_utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -299,8 +299,8 @@
var (
protocolTCP = corev1.ProtocolTCP
adminAPISSLPort = intstr.FromInt(consts.DataPlaneAdminAPIPort)
proxyPort = intstr.FromInt(consts.DataPlaneProxyPort)
proxySSLPort = intstr.FromInt(consts.DataPlaneProxySSLPort)
proxyPort = intstr.FromInt(consts.DataPlaneProxyHTTPPort)
proxySSLPort = intstr.FromInt(consts.DataPlaneProxyHTTPSPort)
metricsPort = intstr.FromInt(consts.DataPlaneMetricsPort)
)

Expand Down Expand Up @@ -527,9 +527,9 @@
return map[gatewayv1.ProtocolType]map[gatewayv1.Kind]struct{}{
gatewayv1.HTTPProtocolType: {"HTTPRoute": {}},
gatewayv1.HTTPSProtocolType: {"HTTPRoute": {}},
gatewayv1.TLSProtocolType: {"TLSRoute": {}},

// L4 routes not supported yet
// gatewayv1.TLSProtocolType: {"TLSRoute": {}},
// TCP and UDP routes not supported yet
// gatewayv1.TCPProtocolType: {"TCPRoute": {}},
// gatewayv1.UDPProtocolType: {"UDPRoute": {}},
}
Expand Down Expand Up @@ -590,10 +590,25 @@
LastTransitionTime: metav1.Now(),
ObservedGeneration: g.Generation,
}
if listener.Protocol != gatewayv1.HTTPProtocolType && listener.Protocol != gatewayv1.HTTPSProtocolType {
if listener.Protocol != gatewayv1.HTTPProtocolType &&
listener.Protocol != gatewayv1.HTTPSProtocolType &&
listener.Protocol != gatewayv1.TLSProtocolType {
Comment on lines +593 to +595
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the supportedRoutesByProtocol() like so:

_, ok := supportedRoutesByProtocol()[listener.Protocol]

and then check !ok? This way we can avoid forgetting to update this when we add support for other protocols. WDYT?

acceptedCondition.Status = metav1.ConditionFalse
acceptedCondition.Reason = string(gatewayv1.ListenerReasonUnsupportedProtocol)
}
// Only TLS terminate mode supported with HTTPS Listeners.
if listener.Protocol == gatewayv1.HTTPSProtocolType && *listener.TLS.Mode != gatewayv1.TLSModeTerminate {
acceptedCondition.Status = metav1.ConditionFalse
acceptedCondition.Reason = string(ListenereReasonInvalidTLSMode)
acceptedCondition.Message = "Only Terminate mode is supported with HTTPS listeners"
}

// Only TLS passthrough mode supported with TLS Listeners.
if listener.Protocol == gatewayv1.TLSProtocolType && *listener.TLS.Mode != gatewayv1.TLSModePassthrough {
acceptedCondition.Status = metav1.ConditionFalse
acceptedCondition.Reason = string(ListenereReasonInvalidTLSMode)
acceptedCondition.Message = "Only Passthrough mode is supported with TLS listeners"
}
Comment on lines +599 to +611
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about using a switch on the protocol here and then nesting the particular conditions for those protocols inside the cases?

listenerConditionsAware := listenerConditionsAware(&g.Status.Listeners[i])
listenerConditionsAware.SetConditions(append(listenerConditionsAware.Conditions, acceptedCondition))
}
Expand Down Expand Up @@ -685,9 +700,11 @@
}
switch l.Protocol {
case gatewayv1.HTTPSProtocolType:
port.TargetPort = intstr.FromInt(consts.DataPlaneProxySSLPort)
port.TargetPort = intstr.FromInt(consts.DataPlaneProxyHTTPSPort)
case gatewayv1.HTTPProtocolType:
port.TargetPort = intstr.FromInt(consts.DataPlaneProxyPort)
port.TargetPort = intstr.FromInt(consts.DataPlaneProxyHTTPPort)
case gatewayv1.TLSProtocolType:
port.TargetPort = intstr.FromInt(consts.DataPlaneProxyTLSPort)
default:
errs = errors.Join(errs, fmt.Errorf("listener %d uses unsupported protocol %s", i, l.Protocol))
continue
Expand All @@ -712,67 +729,66 @@

message := ""
if listener.TLS != nil {
// We currently do not support TLSRoutes, hence only TLS termination supported.
if *listener.TLS.Mode != gatewayv1.TLSModeTerminate {
resolvedRefsCondition.Status = metav1.ConditionFalse
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, "Only Terminate mode is supported")
}
// We currently do not support more that one listener certificate.
if len(listener.TLS.CertificateRefs) != 1 {
if len(listener.TLS.CertificateRefs) > 1 {
resolvedRefsCondition.Reason = string(ListenerReasonTooManyTLSSecrets)
message = conditionMessage(message, "Only one certificate per listener is supported")
} else {

Check failure on line 736 in controller/gateway/controller_reconciler_utils.go

View workflow job for this annotation

GitHub Actions / lint

elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic)
isValidGroupKind := true
certificateRef := listener.TLS.CertificateRefs[0]
if certificateRef.Group != nil && *certificateRef.Group != "" && *certificateRef.Group != gatewayv1.Group(corev1.SchemeGroupVersion.Group) {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, fmt.Sprintf("Group %s not supported in CertificateRef", *certificateRef.Group))
isValidGroupKind = false
}
if certificateRef.Kind != nil && *certificateRef.Kind != "" && *certificateRef.Kind != gatewayv1.Kind("Secret") {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, fmt.Sprintf("Kind %s not supported in CertificateRef", *certificateRef.Kind))
isValidGroupKind = false
}
secretNamespace := gatewayNamespace
if certificateRef.Namespace != nil && *certificateRef.Namespace != "" {
secretNamespace = string(*certificateRef.Namespace)
}

var secretExists bool
if isValidGroupKind {
// Get the secret and check it exists.
certificateSecret := &corev1.Secret{}
err = c.Get(ctx, types.NamespacedName{
Namespace: secretNamespace,
Name: string(certificateRef.Name),
}, certificateSecret)
if err != nil {
if !k8serrors.IsNotFound(err) {
return
}
// check certificate references only when Terminate mode is used.
// Passthrough mode does not need a certificate.
if len(listener.TLS.CertificateRefs) != 0 {
isValidGroupKind := true
certificateRef := listener.TLS.CertificateRefs[0]
if certificateRef.Group != nil && *certificateRef.Group != "" && *certificateRef.Group != gatewayv1.Group(corev1.SchemeGroupVersion.Group) {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, fmt.Sprintf("Referenced secret %s/%s does not exist", secretNamespace, certificateRef.Name))
} else {
secretExists = true
message = conditionMessage(message, fmt.Sprintf("Group %s not supported in CertificateRef", *certificateRef.Group))
isValidGroupKind = false
}
if certificateRef.Kind != nil && *certificateRef.Kind != "" && *certificateRef.Kind != gatewayv1.Kind("Secret") {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, fmt.Sprintf("Kind %s not supported in CertificateRef", *certificateRef.Kind))
isValidGroupKind = false
}
secretNamespace := gatewayNamespace
if certificateRef.Namespace != nil && *certificateRef.Namespace != "" {
secretNamespace = string(*certificateRef.Namespace)
}
}

if secretExists {
// In case there is a cross-namespace reference, check if there is any referenceGrant allowing it.
if secretNamespace != gatewayNamespace {
referenceGrantList := &gatewayv1beta1.ReferenceGrantList{}
err = c.List(ctx, referenceGrantList, client.InNamespace(secretNamespace))
var secretExists bool
if isValidGroupKind {
// Get the secret and check it exists.
certificateSecret := &corev1.Secret{}
err = c.Get(ctx, types.NamespacedName{
Namespace: secretNamespace,
Name: string(certificateRef.Name),
}, certificateSecret)
if err != nil {
return
if !k8serrors.IsNotFound(err) {
return
}
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonInvalidCertificateRef)
message = conditionMessage(message, fmt.Sprintf("Referenced secret %s/%s does not exist", secretNamespace, certificateRef.Name))
} else {
secretExists = true
}
if !isSecretCrossReferenceGranted(gatewayv1.Namespace(gatewayNamespace), certificateRef.Name, referenceGrantList.Items) {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonRefNotPermitted)
message = conditionMessage(message, fmt.Sprintf("Secret %s/%s reference not allowed by any referenceGrant", secretNamespace, certificateRef.Name))
}

if secretExists {
// In case there is a cross-namespace reference, check if there is any referenceGrant allowing it.
if secretNamespace != gatewayNamespace {
referenceGrantList := &gatewayv1beta1.ReferenceGrantList{}
err = c.List(ctx, referenceGrantList, client.InNamespace(secretNamespace))
if err != nil {
return
}
if !isSecretCrossReferenceGranted(gatewayv1.Namespace(gatewayNamespace), certificateRef.Name, referenceGrantList.Items) {
resolvedRefsCondition.Reason = string(gatewayv1.ListenerReasonRefNotPermitted)
message = conditionMessage(message, fmt.Sprintf("Secret %s/%s reference not allowed by any referenceGrant", secretNamespace, certificateRef.Name))
}
}
}
}

}
}

Expand Down
42 changes: 3 additions & 39 deletions controller/gateway/controller_reconciler_utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -553,12 +553,12 @@ func TestSetDataPlaneIngressServicePorts(t *testing.T) {
{
Name: "http",
Port: 80,
TargetPort: intstr.FromInt(consts.DataPlaneProxyPort),
TargetPort: intstr.FromInt(consts.DataPlaneProxyHTTPPort),
},
{
Name: "https",
Port: 443,
TargetPort: intstr.FromInt(consts.DataPlaneProxySSLPort),
TargetPort: intstr.FromInt(consts.DataPlaneProxyHTTPSPort),
},
},
},
Expand All @@ -580,7 +580,7 @@ func TestSetDataPlaneIngressServicePorts(t *testing.T) {
{
Name: "http",
Port: 80,
TargetPort: intstr.FromInt(consts.DataPlaneProxyPort),
TargetPort: intstr.FromInt(consts.DataPlaneProxyHTTPPort),
},
},
expectedError: errors.New("listener 1 uses unsupported protocol UDP"),
Expand Down Expand Up @@ -979,42 +979,6 @@ func TestGetSupportedKindsWithResolvedRefsCondition(t *testing.T) {
ObservedGeneration: generation,
},
},
{
name: "tls with passthrough, HTTPS protocol, no allowed routes",
gatewayNamespace: "default",
listener: gatewayv1.Listener{
Protocol: gatewayv1.HTTPSProtocolType,
TLS: &gatewayv1.GatewayTLSConfig{
Mode: lo.ToPtr(gatewayv1.TLSModePassthrough),
CertificateRefs: []gatewayv1.SecretObjectReference{
{
Name: "test-secret",
},
},
},
},
secrets: []client.Object{
&corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: "test-secret",
Namespace: "default",
},
},
},
expectedSupportedKinds: []gatewayv1.RouteGroupKind{
{
Group: (*gatewayv1.Group)(&gatewayv1.GroupVersion.Group),
Kind: "HTTPRoute",
},
},
expectedResolvedRefsCondition: metav1.Condition{
Type: string(gatewayv1.ListenerConditionResolvedRefs),
Status: metav1.ConditionFalse,
Reason: string(gatewayv1.ListenerReasonInvalidCertificateRef),
Message: "Only Terminate mode is supported.",
ObservedGeneration: generation,
},
},
{
name: "tls bad-formed, multiple TLS secrets no cross-namespace reference",
gatewayNamespace: "default",
Expand Down
10 changes: 10 additions & 0 deletions controller/pkg/controlplane/controlplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,16 @@ func SetDefaults(
changed = true
}

// If the feature gates are not set, set them to the default value.
const (
featureGates = "CONTROLLER_FEATURE_GATES"
defaultFeatureGates = "GatewayAlpha=true"
)
if k8sutils.EnvValueByName(container.Env, featureGates) == "" {
container.Env = k8sutils.UpdateEnv(container.Env, featureGates, defaultFeatureGates)
changed = true
}

if args.Namespace != "" && args.DataPlaneIngressServiceName != "" {
if _, isOverrideDisabled := dontOverride["CONTROLLER_PUBLISH_SERVICE"]; !isOverrideDisabled {
publishServiceNN := k8stypes.NamespacedName{Namespace: args.Namespace, Name: args.DataPlaneIngressServiceName}.String()
Expand Down
3 changes: 2 additions & 1 deletion internal/utils/dataplane/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ var KongDefaults = map[string]string{
"KONG_PORT_MAPS": "80:8000, 443:8443",
"KONG_PROXY_ACCESS_LOG": "/dev/stdout",
"KONG_PROXY_ERROR_LOG": "/dev/stderr",
"KONG_PROXY_LISTEN": fmt.Sprintf("0.0.0.0:%d reuseport backlog=16384, 0.0.0.0:%d http2 ssl reuseport backlog=16384", consts.DataPlaneProxyPort, consts.DataPlaneProxySSLPort),
"KONG_PROXY_LISTEN": fmt.Sprintf("0.0.0.0:%d reuseport backlog=16384, 0.0.0.0:%d http2 ssl reuseport backlog=16384", consts.DataPlaneProxyHTTPPort, consts.DataPlaneProxyHTTPSPort),
"KONG_STATUS_LISTEN": fmt.Sprintf("0.0.0.0:%d", consts.DataPlaneStatusPort),
"KONG_STREAM_LISTEN": fmt.Sprintf("0.0.0.0:%d ssl", consts.DataPlaneProxyTLSPort),

// TODO: reconfigure following https://github.com/Kong/gateway-operator/issues/7
"KONG_ADMIN_LISTEN": fmt.Sprintf("0.0.0.0:%d ssl reuseport backlog=16384", consts.DataPlaneAdminAPIPort),
Expand Down
7 changes: 5 additions & 2 deletions pkg/consts/dataplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,10 +135,13 @@ const (
DataPlaneAdminAPIPort = 8444

// DataPlaneHTTPSPort is the port that the dataplane uses for HTTP.
DataPlaneProxyPort = 8000
DataPlaneProxyHTTPPort = 8000

// DataPlaneHTTPSPort is the port that the dataplane uses for HTTPS.
DataPlaneProxySSLPort = 8443
DataPlaneProxyHTTPSPort = 8443

// DataPlaneProxyTLSPort is the port that the dataplane uses for TLS.
DataPlaneProxyTLSPort = 9443

// DataPlaneMetricsPort is the port that the dataplane uses for metrics.
DataPlaneMetricsPort = 8100
Expand Down
9 changes: 7 additions & 2 deletions pkg/utils/kubernetes/resources/deployments.go
Original file line number Diff line number Diff line change
Expand Up @@ -295,12 +295,17 @@ func GenerateDataPlaneContainer(image string) corev1.Container {
Ports: []corev1.ContainerPort{
{
Name: "proxy",
ContainerPort: consts.DataPlaneProxyPort,
ContainerPort: consts.DataPlaneProxyHTTPPort,
Protocol: corev1.ProtocolTCP,
},
{
Name: "proxy-ssl",
ContainerPort: consts.DataPlaneProxySSLPort,
ContainerPort: consts.DataPlaneProxyHTTPSPort,
Protocol: corev1.ProtocolTCP,
},
{
Name: "proxy-tls",
ContainerPort: consts.DataPlaneProxyTLSPort,
Protocol: corev1.ProtocolTCP,
},
{
Expand Down
6 changes: 3 additions & 3 deletions pkg/utils/kubernetes/resources/services.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,13 @@ var DefaultDataPlaneIngressServicePorts = []corev1.ServicePort{
Name: "http",
Protocol: corev1.ProtocolTCP,
Port: consts.DefaultHTTPPort,
TargetPort: intstr.FromInt(consts.DataPlaneProxyPort),
TargetPort: intstr.FromInt(consts.DataPlaneProxyHTTPPort),
},
{
Name: "https",
Protocol: corev1.ProtocolTCP,
Port: consts.DefaultHTTPSPort,
TargetPort: intstr.FromInt(consts.DataPlaneProxySSLPort),
TargetPort: intstr.FromInt(consts.DataPlaneProxyHTTPSPort),
},
}

Expand Down Expand Up @@ -145,7 +145,7 @@ func ServicePortsFromDataPlaneIngressOpt(dataplane *operatorv1beta1.DataPlane) S
newPorts := make([]corev1.ServicePort, 0)
alreadyUsedPorts := make(map[int32]struct{})
for _, p := range dataplane.Spec.Network.Services.Ingress.Ports {
targetPort := intstr.FromInt(consts.DataPlaneProxyPort)
targetPort := intstr.FromInt(consts.DataPlaneProxyHTTPPort)
if !cmp.Equal(p.TargetPort, intstr.IntOrString{}) {
targetPort = p.TargetPort
}
Expand Down
Loading
Loading