With External Access & external-dns enabled, internal clients should be able to connect to segment store without external DNS resolution

[pbelgundi]

Description

When we setup Pravega with External Access and external-dns enabled, segment stores advertise external-dns endpoint to clients. As such, internal pravega clients (in the same K8s cluster as pravega deployment) also get routed via external-dns resolution, when connecting to segment store instances. See related issue #3899. This needs to be avoided so internal clients can directly connect to SegmentStore without going through the external dns resolution.

Importance

Should-have: Optimize access from internal clients

Suggestions for an improvement

One way to implement this can be to have the same dns entry as in external dns server inside kubernetes internal dns server (coredns).

[EronWright]

@pbelgundi with the suggestion to use DNS to achieve the desired effect, one would need to be careful about SSL hostname verification.

My suggestion is to advertise numerous endpoints per segment store. For example, a given segment store could expose a set of labeled endpoints. The ZK data would resemble:

endpoints:
  segmentstore-1:
    internal: http://ip-172-31-18-160.us-west-2.compute.internal:80
    external: https://ec2-54-191-84-122.us-west-2.compute.amazonaws.com:443

I would suggest that the schema and port vary by endpoint.

On the client, one may use an explicit approach to configure which endpoint to use, e.g. ClientConfig.builder().listener("external"). And/or use a discovery approach where the client attempts to connect to each endpoint in turn.