Blue/Green Deployment?

[DavidPx]

Hi all, as part of creating a proposal for using Elsa in our organization I'm figuring out production deployment scenarios. All of our API-style services follow the blue/green pattern. Any kind of queue-consuming application (i.e. RabbitMQ listener) is deployed in a simpler, destroy/deploy manner.

My proposed Elsa design would be deployed as an API hosting Elsa, thus creating a hybrid scenario: an API service which also works off a queue, the queue in this case being the Elsa DB. As I was playing with an experimental deployment I quickly discovered that the inactive instance would conflict with the live instance because they both work off the same database.

I'm now developing a simpler deployment system which maintains only a single instance of the API, destroying and then creating the instance. The downside to this approach is well, downtime on the API. I see no problems with stopping Elsa from scanning for runnable workflows but API downtime will be a harder pill to swallow.

Does anyone have any advice for this situation? Perhaps I should not be combining API concerns with "backend" concerns?

Thank you!

[sfmskywalker]

Hi @DavidPx,

If I understand correctly, the issue is that the "blue" instance is supposed to be inactive (the router won't be sending traffic there), but this instance is still processing work from the database together with the "green" instance.

You say "conflicts" - what does that mean concretely? From a scaling point of view, Elsa is designed to allow for more than one instance to run, all of them pointing to the same database. Perhaps you simply want to prevent the blue instance from running workflows, because it is supposed to be inactive?

If so, then it seems to me that ideally, you can somehow "stop" all this background work of the inactive instance. Unfortunately there's no simple switch for this I don't think, since various background tasks are started when the instance starts. It might be possible to provide a way to programmatically control this, but we don't have that today.

Another approach might be to host Elsa separate from your API, and have your own API interact with Elsa's API endpoints instead.
This way, you can still run any number of Elsa instances to ensure redundancy and increased throughput, and have your own API instances rotate as you see fit using blue/green deployment.

[DavidPx]

Thanks for your thoughts on this!

Your assumption is correct. My concrete example is kind of funny - while messing with the deployment I started both instances at the same time over an empty database - they both tried to run the EF migrations with the expected "table already exists"-style outcomes 🤦 A more realistic example would be the inactive instance (being the previous version), still having a bug in its implementation, picking up runnable instances and causing problems.

If Elsa had a "write only" switch you could get good uptime between the two instances. In this mode the framework would only accept new workflow instances, events, etc. It would not poll for runnable instances. In my head the sequence would look like this. Instance A is the active one, B is inactive.

1. Set instance A to write-only
2. Start deployment of instance B
3. Wait for instance B to be healthy
4. DNS flip