Eureka 0-downtime Blue/Green deployments issue

[leonard520]

Describe the bug
We are leveraging eureka for the service discovery function and we have two different deployments for one app as blue-green deployments.
At beginning, deployment A is start up as eureka.instance.initialStatus=UP while deployment B as eureka.instance.initialStatus=OUT_OF_SERVICE. After running sometime, we switch them by PUT /eureka/v2/apps/appID/instanceID/status?value={OUT_OF_SERVICE or UP}. It works good usually.
However, sometimes, we find eureka server is blocked by network issue to this app, and the lease for the app expires. Then eureka server will ask client to re-register. In this case, the initialized status is not correct for the two deployments.
Is there a way that we can override “eureka.instance.initial-status” without restarting these instances? Such as config hot reload? Will it work since app is up and running for a long time.

[leonard520]

I find that the initial status can be refreshed when EurekaInstanceConfigBean is reloaded.
So one possible solution is to use Config Server to store eureka.instance.initialStatus and when want to swap the two deployments,

1. update the value in Config Server
2. curl -XPOST http://{eureka-client}/actuator/refresh together with overriding the status to the eureka server-side so that they can both update to the new status.

This is a bit complicated, not sure if any other methods can achieve it.

[spencergibb]

You can use the ServiceRegistry.setStatus() method. https://github.com/spring-cloud/spring-cloud-netflix/blob/main/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/serviceregistry/EurekaServiceRegistry.java#L75

[leonard520]

@spencergibb Thanks for the response. May I double confirm that the operation ServiceRegistry.setStatus() happens in client app side? If that case, when should the client app set the value? Does it mean the client app should be aware of the time the swap of blue-green happens?

[spencergibb]

ServiceRegistry.setStatus() can happen anywhere. It's just calling the eureka server API. I'd say that something has to be aware of the blue/green swap, but not necessarily a client.

[leonard520]

@spencergibb I think the problem we found is caused by the obsolete client status. Even the server-side is aware of the blue/green swap, when the client lost connection due to a network issue or so, the client will re-registry with a wrong status.

[spencergibb]

Something has to change the status of that client, there's no other way around it.

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[leonard520]

Do we have a plan to improve this behavior that the client can be notified about the change? I believe this is a common issue for Eureka users.

[gzmrst-sr]

We are running into this problem as a user of a PaaS with a provided Eureka server.
The PaaS does blue/green deployments by setting the status as described by @leonard520 above...
The application being deployed has no knowledge of how it is deployed or of any change it needs to make in the status.

What should be the intended behaviour in situations like this? Do we have to customise the client side Eureka client code to change the initial status? Are there any plans to fix blue/green deployments with Eureka?

[spring-cloud-issues]

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

[gzmrst-sr]

We are still struggling with this issue.
It happens whenever the app loses connection to Eureka for a time, regardless of the cause.

@spencergibb Could you please answer my question above?
How is it supposed to work?

Or is it the case that the initial status was not intended for use like this? (Although I have trouble seeing where else it would be used)

In any case, the problem is that the Eureka client is liable to re-register itself with the initial status every time it loses connection. And there is no mechanism to change the initial status that I am aware of.

[gzmrst-sr]

@spencergibb @OlgaMaciaszek Apologies for bumping an old issue, but it is still a problem for us and we'd greatly appreciate some pointers on how it's supposed to work and/or how we should work around the problem.
Right now we have to just accept occasional outages caused by an incorrect Eureka status and restart everything, which seems sub optimal to say the least.

[spring-cloud-issues]

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

[gzmrst-sr]

@spring-cloud-issues @OlgaMaciaszek Yes please, we would still like to have our questions around this issue answered...

[spring-cloud-issues]

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

[spring-cloud-issues]

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.