Final switch to ElasticSearch from Solr

[mlissner]

We promised that we'd deprecate the Solr version of v3 of the API and webhooks on November 25th.

In #4650, we learned that a brown out strategy should be pretty easy, so this ticket is to discuss the full profess we need to do on Monday and through the end of the week.

There are three waffle flags that we need to flip:

• The first flag will make v1 of the webhooks use Elastcic. This can be flipped on the 25th.
• The second flag will make the front end use Elastic. This can be flipped on the 25th.
• The third flag will make v3 of the API use Elastic. This should be exponentially browned-out:
  • Monday: 5%
  • Tuesday: 10%
  • Wednesday: 25%
  • Thursday: 50%
  • Friday: 100%

By Friday when this is done, Solr can probably be turned off. We'll see. Alberto, can you please chime in with the details and pieces I've missed about this?

[albertisfu]

By Friday when this is done, Solr can probably be turned off. We'll see. Alberto, can you please chime in with the details and pieces I've missed about this?

Sure, here are some details about this process.

Switch V3 of RECAP Search API to ES

To begin serving requests via ES, we can set the flag r-es-search-api-active to 5% on Monday.
r-es-search-api-active

Breaking changes outlined in the migration guide for type=r will begin applying to requests randomly, based on the percentage set.

Switch V3 of Opinion Search API to ES

This process should follow these steps:

• We should do the switch once 4657 Allow users to select the preferred webhook version #4689 and 4682 Updated cl_send_alerts to support V2 webhooks while maintaining compatibility with V1 webhooks #4705 are merged so users can chose their preferred Opinion Search Alert webhook version.

• Run the following command to assess how many manual tweaks are needed to resolve incompatible Opinion alerts for ES:

manage.py clean_up_search_alerts --action validate-queries --validation-wait 1

If errors are found, they are likely to involve syntax issues, such as unbalanced quotes, parentheses, or invalid characters in ES queries.
We expect only a few errors, which can be manually reviewed and corrected in the admin interface.

If we have time it'd be a good idea to run this command before Monday in order to estimate the manual effort required. If many alerts share common issues, we could potentially write a script to fix them in bulk.

After fixes are in place we should run the command again to confirm all its correct now.

If no errors remain, then we'll need to execute the following command to replace the stat_ filter in current alerts with the new values for ES:

manage.py clean_up_search_alerts --action clean-up

Immediately after making this change, enable the following flag and switch for everyone:

o-es-active
o-es-alerts-active

So the Opinion Search frontend and also Opinion Search Alerts start working with ES.

Something to consider is that when we turn enable o-es-active it'll start using ES will be used to query Related Cases on the Opinions page and we still need to verify that this functionality works correctly or apply any necessary tweaks, as planned in Issue #4305.

However I think this issue is planned for next sprint? If so we could introduce a separate flag now to ensure this feature isn't enabled when o-es-active is turned enabled for other frontend functionalities.

Finally, set a percentage for the o-es-search-api-active flag to start serving V3 Opinion Search Alert API requests via ES.

[mlissner]

I think this is done, right, Alberto?

[albertisfu]

Yeah this is done, we can close it.