How to deploy high availability?

[beelzebub2006]

How to deploy high availability?

[rvowles]

@beelzebub2006 Hi there! High availability for SDK consumers means ensuring that Edge and Dacha are available - so you would normally need at least 3 of those, one in each AZ. High availability for your users means multiple MR instances and a highly reliable replicated database. The Kubernetes configuration would normally give you this (except the database) as is. Did you have anything more specific in mind?

[seany89]

@rvowles
We currently have FH deployed on a centralized cluster, with other clusters connecting to the edge nodes through ALB.

We want to make this more HA, is it possible to also have an instance of the Edge service running in the external clusters as well that communicates back to the centralized cluster?

So the app/SDK would just call the edge running local to them, is this possible or have you seen before?

[rvowles]

You would really need to have Dacha in that cluster as well, because Edge stores no state and relies on Dacha. Dacha2 if the state is missing will make a call back to MR (which can be protected by configuration - it would allow you to expose it securely).

But to get Dacha to be kept up to date, you could need to configure one or more NATS servers (presuming you aren't using Google PubSub) to connect back to your main cluster as well, because both Dacha and Edge rely on NATs for streaming updates.

So the answer is yes, its absolutely possible - NATs is particularly well suited to that use case and running Dacha/Edge once it is up and running is easy.

[seany89]

So we exposed MR on the main cluster and configured Dacha2 on the external cluster to pull its cache.
This seems to work fine.

The next step is to get the streaming updates from the NATS server when a change is made on the main cluster MR.

You mentioned we need to configure one or more NAT servers to connect back to the main cluster. Can we just expose the NATS server on the main cluster using ingress/ALB and then get the external cluster to talk back to it?

Would we just use the value nats.urls ?
Is there any authentication options for this?

[rvowles]

I think you are after NATS Super-clusters? https://docs.nats.io/running-a-nats-service/configuration/gateways - and yes, NATS has a robust authentication and authorisation mechanism. Flexe.com have installed it with full permission management when I last talked to them.

[seany89]

@rvowles
Can you expand upon connectivity options to the MR from Dacha2 you mentioned (which can be protected by configuration - it would allow you to expose it securely) what auth options are there?

We can see it normally just connects locally over the "http://featurehub-management-repository:8701" monitor port.
So we would have to expose this endpoint as an ingress on our centralized cluster?

[rvowles]

Yes, but you can put a password on it and it will prevent access without the password. It's in the docs, in the same place as that other config option.

[seany89]

Sorry can you post a link to the docs?

[rvowles]

I appear to have mis-disdocumented it sorry, I have been away from my computer for a couple of days. On the Dacha2 service, you set dacha2.cache.api-key - and on MR you set mr.dacha2.api-keys. You set a list of keys on MR (comma separated if you have more than one) which allows you to rotate the key on the Dacha2 side if you wish.

If you set these, then Dacha2 will add a new "key" header in its call to MR when it is dynamically filling the cache based on requests from edge that aren't in its local cache. MR will compare that key header against the ones you specify and honour the call if so. I will update the documentation and add diagrams.

[rvowles]

I created a PR for this docs update https://github.com/featurehub-io/featurehub/pull/1086/files

[joelp172]

I raised an issue ref setting the nats.urls in a secure manner. As currently it means exposing username and password in plain text

featurehub-io/featurehub-helm#14