diff --git a/README.md b/README.md index cbfa945647..7aab4b1a1f 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,7 @@ The following tutorials are provided: * [Route53](docs/tutorials/aws.md) * [Same domain for public and private Route53 zones](docs/tutorials/public-private-route53.md) * [Cloud Map](docs/tutorials/aws-sd.md) + * [Kube Ingress AWS Controller](docs/tutorials/kube-ingress-aws.md) * [Azure DNS](docs/tutorials/azure.md) * [Azure Private DNS](docs/tutorials/azure-private-dns.md) * [Cloudflare](docs/tutorials/cloudflare.md) diff --git a/docs/tutorials/kube-ingress-aws.md b/docs/tutorials/kube-ingress-aws.md new file mode 100644 index 0000000000..b0f176bc49 --- /dev/null +++ b/docs/tutorials/kube-ingress-aws.md @@ -0,0 +1,307 @@ +# Using ExternalDNS with kube-ingress-aws-controller + +This tutorial describes how to use ExternalDNS with the [kube-ingress-aws-controller][1]. + +[1]: https://github.com/zalando-incubator/kube-ingress-aws-controller + +## Setting up ExternalDNS and kube-ingress-aws-controller + +Follow the [AWS tutorial](aws.md) to setup ExternalDNS for use in Kubernetes clusters +running in AWS. Specify the `source=ingress` argument so that ExternalDNS will look +for hostnames in Ingress objects. In addition, you may wish to limit which Ingress +objects are used as an ExternalDNS source via the `ingress-class` argument, but +this is not required. + +For help setting up the Kubernetes Ingress AWS Controller, that can +create ALBs and NLBs, follow the [Setup Guide][2]. + +[2]: https://github.com/zalando-incubator/kube-ingress-aws-controller/tree/master/deploy + + +### Optional RouteGroup + +[RouteGroup][3] is a CRD, that enables you to do complex routing with +[Skipper][4]. + +First, you have to apply the RouteGroup CRD to your cluster: + +``` +kubectl apply -f https://github.com/zalando/skipper/blob/master/dataclients/kubernetes/deploy/apply/routegroups_crd.yaml +``` + +You have to grant all controllers: [Skipper][4], +[kube-ingress-aws-controller][1] and external-dns to read the routegroup resource and +kube-ingress-aws-controller to update the status field of a routegroup. +This depends on your RBAC policies, in case you use RBAC, you can use +this for all 3 controllers: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1beta1 +kind: ClusterRole +metadata: + name: kube-ingress-aws-controller +rules: +- apiGroups: + - extensions + resources: + - ingresses + verbs: + - get + - list + - watch +- apiGroups: + - extensions + resources: + - ingresses/status + verbs: + - patch + - update +- apiGroups: + - zalando.org + resources: + - routegroups + verbs: + - get + - list + - watch +- apiGroups: + - zalando.org + resources: + - routegroups/status + verbs: + - patch + - update +``` + +See also current RBAC yaml files: +- [kube-ingress-aws-controller](https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/ingress-controller/rbac.yaml) +- [skipper](https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/skipper/rbac.yaml) +- [external-dns](https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/external-dns/rbac.yaml) + +[3]: https://opensource.zalando.com/skipper/kubernetes/routegroups/#routegroups +[4]: https://opensource.zalando.com/skipper + + +## Deploy an example application + +Create the following sample "echoserver" application to demonstrate how +ExternalDNS works with ingress objects, that were created by [kube-ingress-aws-controller][1]. + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: echoserver +spec: + replicas: 1 + selector: + matchLabels: + app: echoserver + template: + metadata: + labels: + app: echoserver + spec: + containers: + - image: gcr.io/google_containers/echoserver:1.4 + imagePullPolicy: Always + name: echoserver + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: echoserver +spec: + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + type: ClusterIP + selector: + app: echoserver +``` + +Note that the Service object is of type `ClusterIP`, because we will +target [Skipper][4] and do the HTTP routing in Skipper. We don't need +a Service of type `LoadBalancer` here, since we will be using a shared +skipper-ingress for all Ingress. Skipper use `hostNetwork` to be able +to get traffic from AWS LoadBalancers EC2 network. ALBs or NLBs, will +be created based on need and will be shared across all ingress as +default. + +## Ingress examples + +Create the following Ingress to expose the echoserver application to the Internet. + +```yaml +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + annotations: + kubernetes.io/ingress.class: skipper + name: echoserver +spec: + rules: + - host: echoserver.mycluster.example.org + http: &echoserver_root + paths: + - backend: + serviceName: echoserver + servicePort: 80 + path: / + - host: echoserver.example.org + http: *echoserver_root +``` + +The above should result in the creation of an (ipv4) ALB in AWS which will forward +traffic to skipper which will forward to the echoserver application. + +If the `--source=ingress` argument is specified, then ExternalDNS will create DNS +records based on the hosts specified in ingress objects. The above example would +result in two alias records being created, `echoserver.mycluster.example.org` and +`echoserver.example.org`, which both alias the ALB that is associated with the +Ingress object. + +Note that the above example makes use of the YAML anchor feature to avoid having +to repeat the http section for multiple hosts that use the exact same paths. If +this Ingress object will only be fronting one backend Service, we might instead +create the following: + +```yaml +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + annotations: + external-dns.alpha.kubernetes.io/hostname: echoserver.mycluster.example.org, echoserver.example.org + kubernetes.io/ingress.class: skipper + name: echoserver +spec: + rules: + - http: + paths: + - backend: + serviceName: echoserver + servicePort: 80 + path: / +``` + +In the above example we create a default path that works for any hostname, and +make use of the `external-dns.alpha.kubernetes.io/hostname` annotation to create +multiple aliases for the resulting ALB. + +## Dualstack ALBs + +AWS [supports](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#ip-address-type) both IPv4 and "dualstack" (both IPv4 and IPv6) interfaces for ALBs. +The Kubernetes Ingress AWS controller supports the `alb.ingress.kubernetes.io/ip-address-type` +annotation (which defaults to `ipv4`) to determine this. If this annotation is +set to `dualstack` then ExternalDNS will create two alias records (one A record +and one AAAA record) for each hostname associated with the Ingress object. + + +Example: + +```yaml +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + annotations: + alb.ingress.kubernetes.io/ip-address-type: dualstack + kubernetes.io/ingress.class: skipper + name: echoserver +spec: + rules: + - host: echoserver.example.org + http: + paths: + - backend: + serviceName: echoserver + servicePort: 80 + path: / +``` + +The above Ingress object will result in the creation of an ALB with a dualstack +interface. ExternalDNS will create both an A `echoserver.example.org` record and +an AAAA record of the same name, that each are aliases for the same ALB. + +## NLBs + +AWS has +[NLBs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) +and [kube-ingress-aws-controller][1] is able to create NLBs instead of ALBs. +The Kubernetes Ingress AWS controller supports the `zalando.org/aws-load-balancer-type` +annotation (which defaults to `alb`) to determine this. If this annotation is +set to `nlb` then ExternalDNS will create an NLB instead of an ALB. + +Example: + +```yaml +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + annotations: + zalando.org/aws-load-balancer-type: nlb + kubernetes.io/ingress.class: skipper + name: echoserver +spec: + rules: + - host: echoserver.example.org + http: + paths: + - backend: + serviceName: echoserver + servicePort: 80 + path: / +``` + +The above Ingress object will result in the creation of an NLB. A +successful create, you can observe in the ingress `status` field, that is +written by [kube-ingress-aws-controller][1]: + +```yaml +status: + loadBalancer: + ingress: + - hostname: kube-ing-lb-atedkrlml7iu-1681027139.$region.elb.amazonaws.com +``` + +ExternalDNS will create a A-records `echoserver.example.org`, that +use AWS ALIAS record to automatically maintain IP adresses of the NLB. + +## RouteGroup (optional) + +[Kube-ingress-aws-controller][1], [Skipper][4] and external-dns +support [RouteGroups][3]. External-dns needs to be started with +`--source=skipper-routegroup` parameter in order to work on RouteGroup objects. + +Here we can not show [all RouteGroup +capabilities](https://opensource.zalando.com/skipper/kubernetes/routegroups/), +but we show one simple example with an application and a custom https +redirect. + +```yaml +apiVersion: zalando.org/v1 +kind: RouteGroup +metadata: + name: my-route-group +spec: + backends: + - name: my-backend + type: service + serviceName: my-service + servicePort: 80 + - name: redirectShunt + type: shunt + defaultBackends: + - backendName: my-service + routes: + - pathSubtree: / + - pathSubtree: / + predicates: + - Header("X-Forwarded-Proto", "http") + filters: + - redirectTo(302, "https:") + backends: + - redirectShunt +``` diff --git a/main.go b/main.go index 29e5c262a6..843ff02a34 100644 --- a/main.go +++ b/main.go @@ -91,6 +91,8 @@ func main() { CFUsername: cfg.CFUsername, CFPassword: cfg.CFPassword, ContourLoadBalancerService: cfg.ContourLoadBalancerService, + SkipperRouteGroupVersion: cfg.SkipperRouteGroupVersion, + RequestTimeout: cfg.RequestTimeout, } // Lookup all the selected sources by names and pass them the desired configuration. diff --git a/pkg/apis/externaldns/types.go b/pkg/apis/externaldns/types.go index e1b2f4ce78..7c29aeb6bb 100644 --- a/pkg/apis/externaldns/types.go +++ b/pkg/apis/externaldns/types.go @@ -24,6 +24,7 @@ import ( "github.com/alecthomas/kingpin" "github.com/sirupsen/logrus" + "sigs.k8s.io/external-dns/source" ) const ( @@ -42,6 +43,7 @@ type Config struct { RequestTimeout time.Duration IstioIngressGatewayServices []string ContourLoadBalancerService string + SkipperRouteGroupVersion string Sources []string Namespace string AnnotationFilter string @@ -144,6 +146,7 @@ var defaultConfig = &Config{ RequestTimeout: time.Second * 30, IstioIngressGatewayServices: []string{"istio-system/istio-ingressgateway"}, ContourLoadBalancerService: "heptio-contour/contour", + SkipperRouteGroupVersion: "zalando.org/v1", Sources: nil, Namespace: "", AnnotationFilter: "", @@ -286,8 +289,12 @@ func (cfg *Config) ParseFlags(args []string) error { // Flags related to Contour app.Flag("contour-load-balancer", "The fully-qualified name of the Contour load balancer service. (default: heptio-contour/contour)").Default("heptio-contour/contour").StringVar(&cfg.ContourLoadBalancerService) + // Flags related to Skipper RouteGroup + app.Flag("skipper-routegroup-groupversion", "The resource version for skipper routegroup").Default(source.DefaultRoutegroupVersion).StringVar(&cfg.SkipperRouteGroupVersion) + // Flags related to processing sources - app.Flag("source", "The resource types that are queried for endpoints; specify multiple times for multiple sources (required, options: service, ingress, node, fake, connector, istio-gateway, cloudfoundry, contour-ingressroute, crd, empty)").Required().PlaceHolder("source").EnumsVar(&cfg.Sources, "service", "ingress", "node", "istio-gateway", "cloudfoundry", "contour-ingressroute", "fake", "connector", "crd", "empty") + app.Flag("source", "The resource types that are queried for endpoints; specify multiple times for multiple sources (required, options: service, ingress, node, fake, connector, istio-gateway, cloudfoundry, contour-ingressroute, crd, empty)").Required().PlaceHolder("source").EnumsVar(&cfg.Sources, "service", "ingress", "node", "istio-gateway", "cloudfoundry", "contour-ingressroute", "fake", "connector", "crd", "empty", "skipper-routegroup") + app.Flag("namespace", "Limit sources of endpoints to a specific namespace (default: all namespaces)").Default(defaultConfig.Namespace).StringVar(&cfg.Namespace) app.Flag("annotation-filter", "Filter sources managed by external-dns via annotation using label selector semantics (default: all sources)").Default(defaultConfig.AnnotationFilter).StringVar(&cfg.AnnotationFilter) app.Flag("fqdn-template", "A templated string that's used to generate DNS names from sources that don't define a hostname themselves, or to add a hostname suffix when paired with the fake source (optional). Accepts comma separated list for multiple global FQDN.").Default(defaultConfig.FQDNTemplate).StringVar(&cfg.FQDNTemplate) diff --git a/pkg/apis/externaldns/types_test.go b/pkg/apis/externaldns/types_test.go index 35a331e138..9763992e83 100644 --- a/pkg/apis/externaldns/types_test.go +++ b/pkg/apis/externaldns/types_test.go @@ -33,6 +33,7 @@ var ( KubeConfig: "", RequestTimeout: time.Second * 30, ContourLoadBalancerService: "heptio-contour/contour", + SkipperRouteGroupVersion: "zalando.org/v1", Sources: []string{"service"}, Namespace: "", FQDNTemplate: "", @@ -103,6 +104,7 @@ var ( KubeConfig: "/some/path", RequestTimeout: time.Second * 77, ContourLoadBalancerService: "heptio-contour-other/contour-other", + SkipperRouteGroupVersion: "zalando.org/v2", Sources: []string{"service", "ingress", "connector"}, Namespace: "namespace", IgnoreHostnameAnnotation: true, @@ -199,6 +201,7 @@ func TestParseFlags(t *testing.T) { "--kubeconfig=/some/path", "--request-timeout=77s", "--contour-load-balancer=heptio-contour-other/contour-other", + "--skipper-routegroup-groupversion=zalando.org/v2", "--source=service", "--source=ingress", "--source=connector", @@ -282,80 +285,81 @@ func TestParseFlags(t *testing.T) { title: "override everything via environment variables", args: []string{}, envVars: map[string]string{ - "EXTERNAL_DNS_MASTER": "http://127.0.0.1:8080", - "EXTERNAL_DNS_KUBECONFIG": "/some/path", - "EXTERNAL_DNS_REQUEST_TIMEOUT": "77s", - "EXTERNAL_DNS_CONTOUR_LOAD_BALANCER": "heptio-contour-other/contour-other", - "EXTERNAL_DNS_SOURCE": "service\ningress\nconnector", - "EXTERNAL_DNS_NAMESPACE": "namespace", - "EXTERNAL_DNS_FQDN_TEMPLATE": "{{.Name}}.service.example.com", - "EXTERNAL_DNS_IGNORE_HOSTNAME_ANNOTATION": "1", - "EXTERNAL_DNS_COMPATIBILITY": "mate", - "EXTERNAL_DNS_PROVIDER": "google", - "EXTERNAL_DNS_GOOGLE_PROJECT": "project", - "EXTERNAL_DNS_GOOGLE_BATCH_CHANGE_SIZE": "100", - "EXTERNAL_DNS_GOOGLE_BATCH_CHANGE_INTERVAL": "2s", - "EXTERNAL_DNS_AZURE_CONFIG_FILE": "azure.json", - "EXTERNAL_DNS_AZURE_RESOURCE_GROUP": "arg", - "EXTERNAL_DNS_AZURE_SUBSCRIPTION_ID": "arg", - "EXTERNAL_DNS_CLOUDFLARE_PROXIED": "1", - "EXTERNAL_DNS_CLOUDFLARE_ZONES_PER_PAGE": "20", - "EXTERNAL_DNS_COREDNS_PREFIX": "/coredns/", - "EXTERNAL_DNS_AKAMAI_SERVICECONSUMERDOMAIN": "oooo-xxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx.luna.akamaiapis.net", - "EXTERNAL_DNS_AKAMAI_CLIENT_TOKEN": "o184671d5307a388180fbf7f11dbdf46", - "EXTERNAL_DNS_AKAMAI_CLIENT_SECRET": "o184671d5307a388180fbf7f11dbdf46", - "EXTERNAL_DNS_AKAMAI_ACCESS_TOKEN": "o184671d5307a388180fbf7f11dbdf46", - "EXTERNAL_DNS_INFOBLOX_GRID_HOST": "127.0.0.1", - "EXTERNAL_DNS_INFOBLOX_WAPI_PORT": "8443", - "EXTERNAL_DNS_INFOBLOX_WAPI_USERNAME": "infoblox", - "EXTERNAL_DNS_INFOBLOX_WAPI_PASSWORD": "infoblox", - "EXTERNAL_DNS_INFOBLOX_WAPI_VERSION": "2.6.1", - "EXTERNAL_DNS_INFOBLOX_VIEW": "internal", - "EXTERNAL_DNS_INFOBLOX_SSL_VERIFY": "0", - "EXTERNAL_DNS_INFOBLOX_MAX_RESULTS": "2000", - "EXTERNAL_DNS_OCI_CONFIG_FILE": "oci.yaml", - "EXTERNAL_DNS_INMEMORY_ZONE": "example.org\ncompany.com", - "EXTERNAL_DNS_DOMAIN_FILTER": "example.org\ncompany.com", - "EXTERNAL_DNS_EXCLUDE_DOMAINS": "xapi.example.org\nxapi.company.com", - "EXTERNAL_DNS_PDNS_SERVER": "http://ns.example.com:8081", - "EXTERNAL_DNS_PDNS_API_KEY": "some-secret-key", - "EXTERNAL_DNS_PDNS_TLS_ENABLED": "1", - "EXTERNAL_DNS_RDNS_ROOT_DOMAIN": "lb.rancher.cloud", - "EXTERNAL_DNS_TLS_CA": "/path/to/ca.crt", - "EXTERNAL_DNS_TLS_CLIENT_CERT": "/path/to/cert.pem", - "EXTERNAL_DNS_TLS_CLIENT_CERT_KEY": "/path/to/key.pem", - "EXTERNAL_DNS_ZONE_ID_FILTER": "/hostedzone/ZTST1\n/hostedzone/ZTST2", - "EXTERNAL_DNS_AWS_ZONE_TYPE": "private", - "EXTERNAL_DNS_AWS_ZONE_TAGS": "tag=foo", - "EXTERNAL_DNS_AWS_ASSUME_ROLE": "some-other-role", - "EXTERNAL_DNS_AWS_BATCH_CHANGE_SIZE": "100", - "EXTERNAL_DNS_AWS_BATCH_CHANGE_INTERVAL": "2s", - "EXTERNAL_DNS_AWS_EVALUATE_TARGET_HEALTH": "0", - "EXTERNAL_DNS_AWS_API_RETRIES": "13", - "EXTERNAL_DNS_AWS_PREFER_CNAME": "true", - "EXTERNAL_DNS_POLICY": "upsert-only", - "EXTERNAL_DNS_REGISTRY": "noop", - "EXTERNAL_DNS_TXT_OWNER_ID": "owner-1", - "EXTERNAL_DNS_TXT_PREFIX": "associated-txt-record", - "EXTERNAL_DNS_TXT_CACHE_INTERVAL": "12h", - "EXTERNAL_DNS_INTERVAL": "10m", - "EXTERNAL_DNS_ONCE": "1", - "EXTERNAL_DNS_DRY_RUN": "1", - "EXTERNAL_DNS_EVENTS": "1", - "EXTERNAL_DNS_LOG_FORMAT": "json", - "EXTERNAL_DNS_METRICS_ADDRESS": "127.0.0.1:9099", - "EXTERNAL_DNS_LOG_LEVEL": "debug", - "EXTERNAL_DNS_CONNECTOR_SOURCE_SERVER": "localhost:8081", - "EXTERNAL_DNS_EXOSCALE_ENDPOINT": "https://api.foo.ch/dns", - "EXTERNAL_DNS_EXOSCALE_APIKEY": "1", - "EXTERNAL_DNS_EXOSCALE_APISECRET": "2", - "EXTERNAL_DNS_CRD_SOURCE_APIVERSION": "test.k8s.io/v1alpha1", - "EXTERNAL_DNS_CRD_SOURCE_KIND": "Endpoint", - "EXTERNAL_DNS_RCODEZERO_TXT_ENCRYPT": "1", - "EXTERNAL_DNS_NS1_ENDPOINT": "https://api.example.com/v1", - "EXTERNAL_DNS_NS1_IGNORESSL": "1", - "EXTERNAL_DNS_TRANSIP_ACCOUNT": "transip", - "EXTERNAL_DNS_TRANSIP_KEYFILE": "/path/to/transip.key", + "EXTERNAL_DNS_MASTER": "http://127.0.0.1:8080", + "EXTERNAL_DNS_KUBECONFIG": "/some/path", + "EXTERNAL_DNS_REQUEST_TIMEOUT": "77s", + "EXTERNAL_DNS_CONTOUR_LOAD_BALANCER": "heptio-contour-other/contour-other", + "EXTERNAL_DNS_SKIPPER_ROUTEGROUP_GROUPVERSION": "zalando.org/v2", + "EXTERNAL_DNS_SOURCE": "service\ningress\nconnector", + "EXTERNAL_DNS_NAMESPACE": "namespace", + "EXTERNAL_DNS_FQDN_TEMPLATE": "{{.Name}}.service.example.com", + "EXTERNAL_DNS_IGNORE_HOSTNAME_ANNOTATION": "1", + "EXTERNAL_DNS_COMPATIBILITY": "mate", + "EXTERNAL_DNS_PROVIDER": "google", + "EXTERNAL_DNS_GOOGLE_PROJECT": "project", + "EXTERNAL_DNS_GOOGLE_BATCH_CHANGE_SIZE": "100", + "EXTERNAL_DNS_GOOGLE_BATCH_CHANGE_INTERVAL": "2s", + "EXTERNAL_DNS_AZURE_CONFIG_FILE": "azure.json", + "EXTERNAL_DNS_AZURE_RESOURCE_GROUP": "arg", + "EXTERNAL_DNS_AZURE_SUBSCRIPTION_ID": "arg", + "EXTERNAL_DNS_CLOUDFLARE_PROXIED": "1", + "EXTERNAL_DNS_CLOUDFLARE_ZONES_PER_PAGE": "20", + "EXTERNAL_DNS_COREDNS_PREFIX": "/coredns/", + "EXTERNAL_DNS_AKAMAI_SERVICECONSUMERDOMAIN": "oooo-xxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx.luna.akamaiapis.net", + "EXTERNAL_DNS_AKAMAI_CLIENT_TOKEN": "o184671d5307a388180fbf7f11dbdf46", + "EXTERNAL_DNS_AKAMAI_CLIENT_SECRET": "o184671d5307a388180fbf7f11dbdf46", + "EXTERNAL_DNS_AKAMAI_ACCESS_TOKEN": "o184671d5307a388180fbf7f11dbdf46", + "EXTERNAL_DNS_INFOBLOX_GRID_HOST": "127.0.0.1", + "EXTERNAL_DNS_INFOBLOX_WAPI_PORT": "8443", + "EXTERNAL_DNS_INFOBLOX_WAPI_USERNAME": "infoblox", + "EXTERNAL_DNS_INFOBLOX_WAPI_PASSWORD": "infoblox", + "EXTERNAL_DNS_INFOBLOX_WAPI_VERSION": "2.6.1", + "EXTERNAL_DNS_INFOBLOX_VIEW": "internal", + "EXTERNAL_DNS_INFOBLOX_SSL_VERIFY": "0", + "EXTERNAL_DNS_INFOBLOX_MAX_RESULTS": "2000", + "EXTERNAL_DNS_OCI_CONFIG_FILE": "oci.yaml", + "EXTERNAL_DNS_INMEMORY_ZONE": "example.org\ncompany.com", + "EXTERNAL_DNS_DOMAIN_FILTER": "example.org\ncompany.com", + "EXTERNAL_DNS_EXCLUDE_DOMAINS": "xapi.example.org\nxapi.company.com", + "EXTERNAL_DNS_PDNS_SERVER": "http://ns.example.com:8081", + "EXTERNAL_DNS_PDNS_API_KEY": "some-secret-key", + "EXTERNAL_DNS_PDNS_TLS_ENABLED": "1", + "EXTERNAL_DNS_RDNS_ROOT_DOMAIN": "lb.rancher.cloud", + "EXTERNAL_DNS_TLS_CA": "/path/to/ca.crt", + "EXTERNAL_DNS_TLS_CLIENT_CERT": "/path/to/cert.pem", + "EXTERNAL_DNS_TLS_CLIENT_CERT_KEY": "/path/to/key.pem", + "EXTERNAL_DNS_ZONE_ID_FILTER": "/hostedzone/ZTST1\n/hostedzone/ZTST2", + "EXTERNAL_DNS_AWS_ZONE_TYPE": "private", + "EXTERNAL_DNS_AWS_ZONE_TAGS": "tag=foo", + "EXTERNAL_DNS_AWS_ASSUME_ROLE": "some-other-role", + "EXTERNAL_DNS_AWS_BATCH_CHANGE_SIZE": "100", + "EXTERNAL_DNS_AWS_BATCH_CHANGE_INTERVAL": "2s", + "EXTERNAL_DNS_AWS_EVALUATE_TARGET_HEALTH": "0", + "EXTERNAL_DNS_AWS_API_RETRIES": "13", + "EXTERNAL_DNS_AWS_PREFER_CNAME": "true", + "EXTERNAL_DNS_POLICY": "upsert-only", + "EXTERNAL_DNS_REGISTRY": "noop", + "EXTERNAL_DNS_TXT_OWNER_ID": "owner-1", + "EXTERNAL_DNS_TXT_PREFIX": "associated-txt-record", + "EXTERNAL_DNS_TXT_CACHE_INTERVAL": "12h", + "EXTERNAL_DNS_INTERVAL": "10m", + "EXTERNAL_DNS_ONCE": "1", + "EXTERNAL_DNS_DRY_RUN": "1", + "EXTERNAL_DNS_EVENTS": "1", + "EXTERNAL_DNS_LOG_FORMAT": "json", + "EXTERNAL_DNS_METRICS_ADDRESS": "127.0.0.1:9099", + "EXTERNAL_DNS_LOG_LEVEL": "debug", + "EXTERNAL_DNS_CONNECTOR_SOURCE_SERVER": "localhost:8081", + "EXTERNAL_DNS_EXOSCALE_ENDPOINT": "https://api.foo.ch/dns", + "EXTERNAL_DNS_EXOSCALE_APIKEY": "1", + "EXTERNAL_DNS_EXOSCALE_APISECRET": "2", + "EXTERNAL_DNS_CRD_SOURCE_APIVERSION": "test.k8s.io/v1alpha1", + "EXTERNAL_DNS_CRD_SOURCE_KIND": "Endpoint", + "EXTERNAL_DNS_RCODEZERO_TXT_ENCRYPT": "1", + "EXTERNAL_DNS_NS1_ENDPOINT": "https://api.example.com/v1", + "EXTERNAL_DNS_NS1_IGNORESSL": "1", + "EXTERNAL_DNS_TRANSIP_ACCOUNT": "transip", + "EXTERNAL_DNS_TRANSIP_KEYFILE": "/path/to/transip.key", }, expected: overriddenConfig, }, diff --git a/source/ingress.go b/source/ingress.go index 59f8d7cf05..040d2e13e7 100644 --- a/source/ingress.go +++ b/source/ingress.go @@ -26,7 +26,6 @@ import ( log "github.com/sirupsen/logrus" "k8s.io/api/extensions/v1beta1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/wait" kubeinformers "k8s.io/client-go/informers" @@ -203,11 +202,7 @@ func (sc *ingressSource) endpointsFromTemplate(ing *v1beta1.Ingress) ([]*endpoin // filterByAnnotations filters a list of ingresses by a given annotation selector. func (sc *ingressSource) filterByAnnotations(ingresses []*v1beta1.Ingress) ([]*v1beta1.Ingress, error) { - labelSelector, err := metav1.ParseToLabelSelector(sc.annotationFilter) - if err != nil { - return nil, err - } - selector, err := metav1.LabelSelectorAsSelector(labelSelector) + selector, err := getLabelSelector(sc.annotationFilter) if err != nil { return nil, err } @@ -220,11 +215,8 @@ func (sc *ingressSource) filterByAnnotations(ingresses []*v1beta1.Ingress) ([]*v filteredList := []*v1beta1.Ingress{} for _, ingress := range ingresses { - // convert the ingress' annotations to an equivalent label selector - annotations := labels.Set(ingress.Annotations) - // include ingress if its annotations match the selector - if selector.Matches(annotations) { + if matchLabelSelector(selector, ingress.Annotations) { filteredList = append(filteredList, ingress) } } diff --git a/source/routegroup.go b/source/routegroup.go new file mode 100644 index 0000000000..51cf2b761e --- /dev/null +++ b/source/routegroup.go @@ -0,0 +1,463 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package source + +import ( + "bytes" + "crypto/tls" + "crypto/x509" + "encoding/json" + "fmt" + "io/ioutil" + "net" + "net/http" + "net/url" + "sort" + "strings" + "sync" + "text/template" + "time" + + log "github.com/sirupsen/logrus" + "sigs.k8s.io/external-dns/endpoint" +) + +const ( + defaultIdleConnTimeout = 30 * time.Second + DefaultRoutegroupVersion = "zalando.org/v1" + routeGroupListResource = "/apis/%s/routegroups" + routeGroupNamespacedResource = "/apis/%s/namespaces/%s/routegroups" +) + +type routeGroupSource struct { + cli routeGroupListClient + master string + namespace string + apiEndpoint string + annotationFilter string + fqdnTemplate *template.Template + combineFQDNAnnotation bool + ignoreHostnameAnnotation bool +} + +// for testing +type routeGroupListClient interface { + getRouteGroupList(string) (*routeGroupList, error) +} + +type routeGroupClient struct { + mu sync.Mutex + quit chan struct{} + client *http.Client + token string + tokenFile string +} + +func newRouteGroupClient(token, tokenPath string, timeout time.Duration) *routeGroupClient { + const ( + tokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token" + rootCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + ) + if tokenPath != "" { + tokenPath = tokenFile + } + + tr := &http.Transport{ + DialContext: (&net.Dialer{ + Timeout: timeout, + KeepAlive: 30 * time.Second, + DualStack: true, + }).DialContext, + TLSHandshakeTimeout: 3 * time.Second, + ResponseHeaderTimeout: timeout, + IdleConnTimeout: defaultIdleConnTimeout, + MaxIdleConns: 5, + MaxIdleConnsPerHost: 5, + } + cli := &routeGroupClient{ + client: &http.Client{ + Transport: tr, + }, + quit: make(chan struct{}), + tokenFile: tokenPath, + token: token, + } + + go func() { + for { + select { + case <-time.After(tr.IdleConnTimeout): + tr.CloseIdleConnections() + cli.updateToken() + case <-cli.quit: + return + } + } + }() + + // in cluster config, errors are treated as not running in cluster + cli.updateToken() + + // cluster internal use custom CA to reach TLS endpoint + rootCA, err := ioutil.ReadFile(rootCAFile) + if err != nil { + return cli + } + certPool := x509.NewCertPool() + if !certPool.AppendCertsFromPEM(rootCA) { + return cli + } + + tr.TLSClientConfig = &tls.Config{ + MinVersion: tls.VersionTLS12, + RootCAs: certPool, + } + + return cli +} + +func (cli *routeGroupClient) updateToken() { + if cli.tokenFile == "" { + return + } + + token, err := ioutil.ReadFile(cli.tokenFile) + if err != nil { + log.Errorf("Failed to read token from file (%s): %v", cli.tokenFile, err) + } + + cli.mu.Lock() + cli.token = string(token) + cli.mu.Unlock() +} + +func (cli *routeGroupClient) getToken() string { + cli.mu.Lock() + defer cli.mu.Unlock() + return cli.token +} + +func (cli *routeGroupClient) getRouteGroupList(url string) (*routeGroupList, error) { + resp, err := cli.get(url) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + return nil, fmt.Errorf("failed to get routegroup list from %s, got: %s", url, resp.Status) + } + + var rgs routeGroupList + err = json.NewDecoder(resp.Body).Decode(&rgs) + if err != nil { + return nil, err + } + + return &rgs, nil +} + +func (cli *routeGroupClient) get(url string) (*http.Response, error) { + req, err := http.NewRequest("GET", url, nil) + if err != nil { + return nil, err + } + return cli.do(req) +} + +func (cli *routeGroupClient) do(req *http.Request) (*http.Response, error) { + if tok := cli.getToken(); tok != "" && req.Header.Get("Authorization") == "" { + req.Header.Set("Authorization", "Bearer "+tok) + } + return cli.client.Do(req) +} + +func parseTemplate(fqdnTemplate string) (tmpl *template.Template, err error) { + if fqdnTemplate != "" { + tmpl, err = template.New("endpoint").Funcs(template.FuncMap{ + "trimPrefix": strings.TrimPrefix, + }).Parse(fqdnTemplate) + } + return tmpl, err +} + +// NewRouteGroupSource creates a new routeGroupSource with the given config. +func NewRouteGroupSource(timeout time.Duration, token, tokenPath, master, namespace, annotationFilter, fqdnTemplate, routegroupVersion string, combineFqdnAnnotation, ignoreHostnameAnnotation bool) (Source, error) { + tmpl, err := parseTemplate(fqdnTemplate) + if err != nil { + return nil, err + } + + if routegroupVersion == "" { + routegroupVersion = DefaultRoutegroupVersion + } + cli := newRouteGroupClient(token, tokenPath, timeout) + + u, err := url.Parse(master) + if err != nil { + return nil, err + } + + apiServer := u.String() + // strip port if well known port, because of TLS certifcate match + if u.Scheme == "https" && u.Port() == "443" { + apiServer = "https://" + u.Hostname() + } + + sc := &routeGroupSource{ + cli: cli, + master: apiServer, + namespace: namespace, + apiEndpoint: apiServer + fmt.Sprintf(routeGroupListResource, routegroupVersion), + annotationFilter: annotationFilter, + fqdnTemplate: tmpl, + combineFQDNAnnotation: combineFqdnAnnotation, + ignoreHostnameAnnotation: ignoreHostnameAnnotation, + } + if namespace != "" { + sc.apiEndpoint = apiServer + fmt.Sprintf(routeGroupNamespacedResource, routegroupVersion, namespace) + } + + log.Infoln("Created route group source") + return sc, nil +} + +// AddEventHandler for routegroup is currently a no op, because we do not implement caching, yet. +func (sc *routeGroupSource) AddEventHandler(func() error, <-chan struct{}, time.Duration) {} + +// Endpoints returns endpoint objects for each host-target combination that should be processed. +// Retrieves all routeGroup resources on all namespaces. +// Logic is ported from ingress without fqdnTemplate +func (sc *routeGroupSource) Endpoints() ([]*endpoint.Endpoint, error) { + rgList, err := sc.cli.getRouteGroupList(sc.apiEndpoint) + if err != nil { + log.Errorf("Failed to get RouteGroup list: %v", err) + return nil, err + } + rgList, err = sc.filterByAnnotations(rgList) + if err != nil { + return nil, err + } + + endpoints := []*endpoint.Endpoint{} + for _, rg := range rgList.Items { + // Check controller annotation to see if we are responsible. + controller, ok := rg.Metadata.Annotations[controllerAnnotationKey] + if ok && controller != controllerAnnotationValue { + log.Debugf("Skipping routegroup %s/%s because controller value does not match, found: %s, required: %s", + rg.Metadata.Namespace, rg.Metadata.Name, controller, controllerAnnotationValue) + continue + } + + eps := sc.endpointsFromRouteGroup(rg) + + if (sc.combineFQDNAnnotation || len(eps) == 0) && sc.fqdnTemplate != nil { + tmplEndpoints, err := sc.endpointsFromTemplate(rg) + if err != nil { + return nil, err + } + + if sc.combineFQDNAnnotation { + eps = append(eps, tmplEndpoints...) + } else { + eps = tmplEndpoints + } + } + + if len(eps) == 0 { + log.Debugf("No endpoints could be generated from routegroup %s/%s", rg.Metadata.Namespace, rg.Metadata.Name) + continue + } + + log.Debugf("Endpoints generated from ingress: %s/%s: %v", rg.Metadata.Namespace, rg.Metadata.Name, eps) + sc.setRouteGroupResourceLabel(rg, eps) + sc.setRouteGroupDualstackLabel(rg, eps) + endpoints = append(endpoints, eps...) + } + + for _, ep := range endpoints { + sort.Sort(ep.Targets) + } + + return endpoints, nil +} + +func (sc *routeGroupSource) endpointsFromTemplate(rg *routeGroup) ([]*endpoint.Endpoint, error) { + // Process the whole template string + var buf bytes.Buffer + err := sc.fqdnTemplate.Execute(&buf, rg) + if err != nil { + return nil, fmt.Errorf("failed to apply template on routegroup %s/%s: %v", rg.Metadata.Namespace, rg.Metadata.Name, err) + } + + hostnames := buf.String() + + // error handled in endpointsFromRouteGroup(), otherwise duplicate log + ttl, _ := getTTLFromAnnotations(rg.Metadata.Annotations) + + targets := getTargetsFromTargetAnnotation(rg.Metadata.Annotations) + + if len(targets) == 0 { + targets = targetsFromRouteGroupStatus(rg.Status) + } + + providerSpecific, setIdentifier := getProviderSpecificAnnotations(rg.Metadata.Annotations) + + var endpoints []*endpoint.Endpoint + // splits the FQDN template and removes the trailing periods + hostnameList := strings.Split(strings.Replace(hostnames, " ", "", -1), ",") + for _, hostname := range hostnameList { + hostname = strings.TrimSuffix(hostname, ".") + endpoints = append(endpoints, endpointsForHostname(hostname, targets, ttl, providerSpecific, setIdentifier)...) + } + return endpoints, nil +} + +func (sc *routeGroupSource) setRouteGroupResourceLabel(rg *routeGroup, eps []*endpoint.Endpoint) { + for _, ep := range eps { + ep.Labels[endpoint.ResourceLabelKey] = fmt.Sprintf("routegroup/%s/%s", rg.Metadata.Namespace, rg.Metadata.Name) + } +} + +func (sc *routeGroupSource) setRouteGroupDualstackLabel(rg *routeGroup, eps []*endpoint.Endpoint) { + val, ok := rg.Metadata.Annotations[ALBDualstackAnnotationKey] + if ok && val == ALBDualstackAnnotationValue { + log.Debugf("Adding dualstack label to routegroup %s/%s.", rg.Metadata.Namespace, rg.Metadata.Name) + for _, ep := range eps { + ep.Labels[endpoint.DualstackLabelKey] = "true" + } + } +} + +// annotation logic ported from source/ingress.go without Spec.TLS part, because it'S not supported in RouteGroup +func (sc *routeGroupSource) endpointsFromRouteGroup(rg *routeGroup) []*endpoint.Endpoint { + endpoints := []*endpoint.Endpoint{} + ttl, err := getTTLFromAnnotations(rg.Metadata.Annotations) + if err != nil { + log.Warnf("Failed to get TTL from annotation: %v", err) + } + + targets := getTargetsFromTargetAnnotation(rg.Metadata.Annotations) + if len(targets) == 0 { + for _, lb := range rg.Status.LoadBalancer.RouteGroup { + if lb.IP != "" { + targets = append(targets, lb.IP) + } + if lb.Hostname != "" { + targets = append(targets, lb.Hostname) + } + } + } + + providerSpecific, setIdentifier := getProviderSpecificAnnotations(rg.Metadata.Annotations) + + for _, src := range rg.Spec.Hosts { + if src == "" { + continue + } + endpoints = append(endpoints, endpointsForHostname(src, targets, ttl, providerSpecific, setIdentifier)...) + } + + // Skip endpoints if we do not want entries from annotations + if !sc.ignoreHostnameAnnotation { + hostnameList := getHostnamesFromAnnotations(rg.Metadata.Annotations) + for _, hostname := range hostnameList { + endpoints = append(endpoints, endpointsForHostname(hostname, targets, ttl, providerSpecific, setIdentifier)...) + } + } + return endpoints +} + +// filterByAnnotations filters a list of routeGroupList by a given annotation selector. +func (sc *routeGroupSource) filterByAnnotations(rgs *routeGroupList) (*routeGroupList, error) { + selector, err := getLabelSelector(sc.annotationFilter) + if err != nil { + return nil, err + } + + // empty filter returns original list + if selector.Empty() { + return rgs, nil + } + + var filteredList []*routeGroup + for _, rg := range rgs.Items { + // include ingress if its annotations match the selector + if matchLabelSelector(selector, rg.Metadata.Annotations) { + filteredList = append(filteredList, rg) + } + } + rgs.Items = filteredList + + return rgs, nil +} + +func targetsFromRouteGroupStatus(status routeGroupStatus) endpoint.Targets { + var targets endpoint.Targets + + for _, lb := range status.LoadBalancer.RouteGroup { + if lb.IP != "" { + targets = append(targets, lb.IP) + } + if lb.Hostname != "" { + targets = append(targets, lb.Hostname) + } + } + + return targets +} + +type routeGroupList struct { + Kind string `json:"kind"` + APIVersion string `json:"apiVersion"` + Metadata routeGroupListMetadata `json:"metadata"` + Items []*routeGroup `json:"items"` +} + +type routeGroupListMetadata struct { + SelfLink string `json:"selfLink"` + ResourceVersion string `json:"resourceVersion"` +} + +type routeGroup struct { + Metadata itemMetadata `json:"metadata"` + Spec routeGroupSpec `json:"spec"` + Status routeGroupStatus `json:"status"` +} + +type itemMetadata struct { + Namespace string `json:"namespace"` + Name string `json:"name"` + Annotations map[string]string `json:"annotations"` +} + +type routeGroupSpec struct { + Hosts []string `json:"hosts"` +} + +type routeGroupStatus struct { + LoadBalancer routeGroupLoadBalancerStatus `json:"loadBalancer"` +} + +type routeGroupLoadBalancerStatus struct { + RouteGroup []routeGroupLoadBalancer `json:"routeGroup"` +} + +type routeGroupLoadBalancer struct { + IP string `json:"ip,omitempty"` + Hostname string `json:"hostname,omitempty"` +} diff --git a/source/routegroup_test.go b/source/routegroup_test.go new file mode 100644 index 0000000000..c8bff1a931 --- /dev/null +++ b/source/routegroup_test.go @@ -0,0 +1,841 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package source + +import ( + "testing" + + "github.com/pkg/errors" + "github.com/stretchr/testify/assert" + "sigs.k8s.io/external-dns/endpoint" +) + +func createTestRouteGroup(ns, name string, annotations map[string]string, hosts []string, destinations []routeGroupLoadBalancer) *routeGroup { + return &routeGroup{ + Metadata: itemMetadata{ + Namespace: ns, + Name: name, + Annotations: annotations, + }, + Spec: routeGroupSpec{ + Hosts: hosts, + }, + Status: routeGroupStatus{ + LoadBalancer: routeGroupLoadBalancerStatus{ + RouteGroup: destinations, + }, + }, + } +} + +func TestEndpointsFromRouteGroups(t *testing.T) { + for _, tt := range []struct { + name string + source *routeGroupSource + rg *routeGroup + want []*endpoint.Endpoint + }{ + { + name: "Empty routegroup should return empty endpoints", + source: &routeGroupSource{}, + rg: &routeGroup{}, + want: []*endpoint.Endpoint{}, + }, + { + name: "Routegroup without hosts and destinations create no endpoints", + source: &routeGroupSource{}, + rg: createTestRouteGroup("namespace1", "rg1", nil, nil, nil), + want: []*endpoint.Endpoint{}, + }, + { + name: "Routegroup without hosts create no endpoints", + source: &routeGroupSource{}, + rg: createTestRouteGroup("namespace1", "rg1", nil, nil, []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }), + want: []*endpoint.Endpoint{}, + }, + { + name: "Routegroup without destinations create no endpoints", + source: &routeGroupSource{}, + rg: createTestRouteGroup("namespace1", "rg1", nil, []string{"rg1.k8s.example"}, nil), + want: []*endpoint.Endpoint{}, + }, + { + name: "Routegroup with hosts and destinations creates an endpoint", + source: &routeGroupSource{}, + rg: createTestRouteGroup("namespace1", "rg1", nil, []string{"rg1.k8s.example"}, []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Routegroup with hostname annotation, creates endpoints from the annotation ", + source: &routeGroupSource{}, + rg: createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + hostnameAnnotationKey: "my.example", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "my.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Routegroup with hosts and destinations and ignoreHostnameAnnotation creates endpoints but ignores annotation", + source: &routeGroupSource{ignoreHostnameAnnotation: true}, + rg: createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + hostnameAnnotationKey: "my.example", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Routegroup with hosts and destinations and ttl creates an endpoint with ttl", + source: &routeGroupSource{ignoreHostnameAnnotation: true}, + rg: createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + ttlAnnotationKey: "2189", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + RecordTTL: endpoint.TTL(2189), + }, + }, + }, + { + name: "Routegroup with hosts and destination IP creates an endpoint", + source: &routeGroupSource{}, + rg: createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + IP: "1.5.1.4", + }, + }, + ), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"1.5.1.4"}), + }, + }, + }, + { + name: "Routegroup with hosts and mixed destinations creates endpoints", + source: &routeGroupSource{}, + rg: createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + IP: "1.5.1.4", + }, + }, + ), + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"1.5.1.4"}), + }, + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }} { + t.Run(tt.name, func(t *testing.T) { + got := tt.source.endpointsFromRouteGroup(tt.rg) + + validateEndpoints(t, got, tt.want) + }) + } + +} + +type fakeRouteGroupClient struct { + returnErr bool + rg *routeGroupList +} + +func (f *fakeRouteGroupClient) getRouteGroupList(string) (*routeGroupList, error) { + if f.returnErr { + return nil, errors.New("Fake route group list error") + } + return f.rg, nil +} + +func TestRouteGroupsEndpoints(t *testing.T) { + for _, tt := range []struct { + name string + source *routeGroupSource + fqdnTemplate string + want []*endpoint.Endpoint + wantErr bool + }{ + { + name: "Empty routegroup should return empty endpoints", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{}, + }, + }, + want: []*endpoint.Endpoint{}, + wantErr: false, + }, + { + name: "Single routegroup should return endpoints", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Single routegroup with combineFQDNAnnotation with fqdn template should return endpoints from fqdnTemplate and routegroup", + fqdnTemplate: "{{.Metadata.Name}}.{{.Metadata.Namespace}}.example", + source: &routeGroupSource{ + combineFQDNAnnotation: true, + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg1.namespace1.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Single routegroup without, with fqdn template should return endpoints from fqdnTemplate", + fqdnTemplate: "{{.Metadata.Name}}.{{.Metadata.Namespace}}.example", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + nil, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.namespace1.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Single routegroup without combineFQDNAnnotation with fqdn template should return endpoints not from fqdnTemplate", + fqdnTemplate: "{{.Metadata.Name}}.{{.Metadata.Namespace}}.example", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "Single routegroup with TTL should return endpoint with TTL", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + ttlAnnotationKey: "2189", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + RecordTTL: endpoint.TTL(2189), + }, + }, + }, + { + name: "Routegroup with hosts and mixed destinations creates endpoints", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + IP: "1.5.1.4", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"1.5.1.4"}), + }, + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "multiple routegroups should return endpoints", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace1", + "rg2", + nil, + []string{"rg2.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace2", + "rg3", + nil, + []string{"rg3.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace3", + "rg", + nil, + []string{"rg.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb2.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg2.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg3.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg.k8s.example", + Targets: endpoint.Targets([]string{"lb2.example.org"}), + }, + }, + }, + { + name: "multiple routegroups with filter annotations should return only filtered endpoints", + source: &routeGroupSource{ + annotationFilter: "kubernetes.io/ingress.class=skipper", + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + "kubernetes.io/ingress.class": "skipper", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace1", + "rg2", + map[string]string{ + "kubernetes.io/ingress.class": "nginx", + }, + []string{"rg2.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace2", + "rg3", + map[string]string{ + "kubernetes.io/ingress.class": "", + }, + []string{"rg3.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace3", + "rg", + nil, + []string{"rg.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb2.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "multiple routegroups with set operation annotation filter should return only filtered endpoints", + source: &routeGroupSource{ + annotationFilter: "kubernetes.io/ingress.class in (nginx, skipper)", + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + "kubernetes.io/ingress.class": "skipper", + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace1", + "rg2", + map[string]string{ + "kubernetes.io/ingress.class": "nginx", + }, + []string{"rg2.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace2", + "rg3", + map[string]string{ + "kubernetes.io/ingress.class": "", + }, + []string{"rg3.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace3", + "rg", + nil, + []string{"rg.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb2.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg2.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }, + { + name: "multiple routegroups with controller annotation filter should not return filtered endpoints", + source: &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + controllerAnnotationKey: controllerAnnotationValue, + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace1", + "rg2", + map[string]string{ + controllerAnnotationKey: "dns", + }, + []string{"rg2.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + createTestRouteGroup( + "namespace2", + "rg3", + nil, + []string{"rg3.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + }, + want: []*endpoint.Endpoint{ + { + DNSName: "rg1.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + { + DNSName: "rg3.k8s.example", + Targets: endpoint.Targets([]string{"lb.example.org"}), + }, + }, + }} { + t.Run(tt.name, func(t *testing.T) { + if tt.fqdnTemplate != "" { + println("fqdnTemplate is set") + tmpl, err := parseTemplate(tt.fqdnTemplate) + if err != nil { + t.Fatalf("Failed to parse template: %v", err) + } + tt.source.fqdnTemplate = tmpl + } + + got, err := tt.source.Endpoints() + if err != nil && !tt.wantErr { + t.Errorf("Got error, but does not want to get an error: %v", err) + } + if tt.wantErr && err == nil { + t.Fatal("Got no error, but we want to get an error") + } + + validateEndpoints(t, got, tt.want) + }) + } +} + +func TestResourceLabelIsSet(t *testing.T) { + source := &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + nil, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + } + + got, _ := source.Endpoints() + for _, ep := range got { + if _, ok := ep.Labels[endpoint.ResourceLabelKey]; !ok { + t.Errorf("Failed to set resource label on ep %v", ep) + } + } +} + +func TestDualstackLabelIsSet(t *testing.T) { + source := &routeGroupSource{ + cli: &fakeRouteGroupClient{ + rg: &routeGroupList{ + Items: []*routeGroup{ + createTestRouteGroup( + "namespace1", + "rg1", + map[string]string{ + ALBDualstackAnnotationKey: ALBDualstackAnnotationValue, + }, + []string{"rg1.k8s.example"}, + []routeGroupLoadBalancer{ + { + Hostname: "lb.example.org", + }, + }, + ), + }, + }, + }, + } + + got, _ := source.Endpoints() + for _, ep := range got { + if v, ok := ep.Labels[endpoint.DualstackLabelKey]; !ok || v != "true" { + t.Errorf("Failed to set resource label on ep %v", ep) + } + } +} + +func TestParseTemplate(t *testing.T) { + for _, tt := range []struct { + name string + annotationFilter string + fqdnTemplate string + combineFQDNAndAnnotation bool + expectError bool + }{ + { + name: "invalid template", + expectError: true, + fqdnTemplate: "{{.Name", + }, + { + name: "valid empty template", + expectError: false, + }, + { + name: "valid template", + expectError: false, + fqdnTemplate: "{{.Name}}-{{.Namespace}}.ext-dns.test.com", + }, + { + name: "valid template", + expectError: false, + fqdnTemplate: "{{.Name}}-{{.Namespace}}.ext-dns.test.com, {{.Name}}-{{.Namespace}}.ext-dna.test.com", + }, + { + name: "valid template", + expectError: false, + fqdnTemplate: "{{.Name}}-{{.Namespace}}.ext-dns.test.com, {{.Name}}-{{.Namespace}}.ext-dna.test.com", + combineFQDNAndAnnotation: true, + }, + { + name: "non-empty annotation filter label", + expectError: false, + annotationFilter: "kubernetes.io/ingress.class=nginx", + }, + } { + t.Run(tt.name, func(t *testing.T) { + _, err := parseTemplate(tt.fqdnTemplate) + if tt.expectError { + assert.Error(t, err) + } else { + assert.NoError(t, err) + } + }) + } +} diff --git a/source/source.go b/source/source.go index 935af8adcc..4588727393 100644 --- a/source/source.go +++ b/source/source.go @@ -24,6 +24,8 @@ import ( "strings" "time" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" "sigs.k8s.io/external-dns/endpoint" ) @@ -209,3 +211,16 @@ func endpointsForHostname(hostname string, targets endpoint.Targets, ttl endpoin return endpoints } + +func getLabelSelector(annotationFilter string) (labels.Selector, error) { + labelSelector, err := metav1.ParseToLabelSelector(annotationFilter) + if err != nil { + return nil, err + } + return metav1.LabelSelectorAsSelector(labelSelector) +} + +func matchLabelSelector(selector labels.Selector, srcAnnotations map[string]string) bool { + annotations := labels.Set(srcAnnotations) + return selector.Matches(annotations) +} diff --git a/source/store.go b/source/store.go index 02db492246..a90e1ded83 100644 --- a/source/store.go +++ b/source/store.go @@ -60,6 +60,8 @@ type Config struct { CFUsername string CFPassword string ContourLoadBalancerService string + SkipperRouteGroupVersion string + RequestTimeout time.Duration } // ClientGenerator provides clients @@ -212,16 +214,24 @@ func BuildWithConfig(source string, p ClientGenerator, cfg *Config) (Source, err return nil, err } return NewCRDSource(crdClient, cfg.Namespace, cfg.CRDSourceKind, scheme) + case "skipper-routegroup": + master := cfg.KubeMaster + tokenPath := "" + token := "" + restConfig, err := GetRestConfig(cfg.KubeConfig, cfg.KubeMaster) + if err == nil { + master = restConfig.Host + tokenPath = restConfig.BearerTokenFile + token = restConfig.BearerToken + } + return NewRouteGroupSource(cfg.RequestTimeout, token, tokenPath, master, cfg.Namespace, cfg.AnnotationFilter, cfg.FQDNTemplate, cfg.SkipperRouteGroupVersion, cfg.CombineFQDNAndAnnotation, cfg.IgnoreHostnameAnnotation) } return nil, ErrSourceNotFound } -// NewKubeClient returns a new Kubernetes client object. It takes a Config and -// uses KubeMaster and KubeConfig attributes to connect to the cluster. If -// KubeConfig isn't provided it defaults to using the recommended default. -func NewKubeClient(kubeConfig, kubeMaster string, requestTimeout time.Duration) (*kubernetes.Clientset, error) { - log.Infof("Instantiating new Kubernetes client") - +// GetRestConfig returns the rest clients config to get automatically +// data if you run inside a cluster or by passing flags. +func GetRestConfig(kubeConfig, kubeMaster string) (*rest.Config, error) { if kubeConfig == "" { if _, err := os.Stat(clientcmd.RecommendedHomeFile); err == nil { kubeConfig = clientcmd.RecommendedHomeFile @@ -246,6 +256,20 @@ func NewKubeClient(kubeConfig, kubeMaster string, requestTimeout time.Duration) return nil, err } + return config, nil +} + +// NewKubeClient returns a new Kubernetes client object. It takes a Config and +// uses KubeMaster and KubeConfig attributes to connect to the cluster. If +// KubeConfig isn't provided it defaults to using the recommended default. +func NewKubeClient(kubeConfig, kubeMaster string, requestTimeout time.Duration) (*kubernetes.Clientset, error) { + log.Infof("Instantiating new Kubernetes client") + config, err := GetRestConfig(kubeConfig, kubeMaster) + if err != nil { + return nil, err + } + + config.Timeout = requestTimeout config.WrapTransport = func(rt http.RoundTripper) http.RoundTripper { return instrumented_http.NewTransport(rt, &instrumented_http.Callbacks{ PathProcessor: func(path string) string { @@ -255,8 +279,6 @@ func NewKubeClient(kubeConfig, kubeMaster string, requestTimeout time.Duration) }) } - config.Timeout = requestTimeout - client, err := kubernetes.NewForConfig(config) if err != nil { return nil, err