[BUG] Errors during simultaneous deployments due to search index race condition

[wbeckler]

Describe the bug
When we have multiple pipelines running at the same time (e.g., during releases), the lambda runs in parallel and we get exceptions thrown when the disassociation does not fine the alias or the old index.

Background: As a part of website and docsite deployment, a script creates a payload of the content of the websites for a custom search. This payload is dumped into OpenSearch using a lambda function written in python. The lambda function dumps the payload into an index and then sets an alias to the new index. It then disassociates the alias from the previous index.

[saimedhi]

Hello @wbeckler, I tried to replicate the bug but I didn't get any error. I tried by uploading search-index.json script repeatedly in s3 bucket (It will trigger pipeline and lambda function). I used sleep(180) in the lamda function by expecting to get parallel threads but I noticed that in my deployment setup, events are running one after the other. second task is only starting after first task execution is done.

[saimedhi]

Can someone please comment what is the actual error you are facing? Like do you have any screenshots? And also can someone show me your deployment setup like pipeline, lambda function, s3 so that I can see how different is it from mine?

[macohen]

@saimedhi thanks for the PR; can you speak to how you reached this solution and your steps for reproducing the bug? I don't see any tests in the PR which could be the clearest way to show how this fixes the problem.