Mute all flaky IndicesRequestCacheIT tests

[andrross]

These are very frequent offenders and are causing a lot of pain in CI.

Related issues:

• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374
• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308
• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testStaleKeysCleanupWithLowThreshold {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"true"}} is flaky #13540
• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheCleanupOnEqualStalenessAndThreshold is flaky #13503
• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheCleanupWithDefaultSettings {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky #13711
• [BUG] org.opensearch.indices.IndicesRequestCacheIT.testDynamicStalenessThresholdUpdate is flaky #13949
• [BUG] Flaky test - org.opensearch.indices.IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices #13437
• [BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[andrross]

@sgup432 @kiranprakash154 @getsaurabh02 Just FYI. It seems likely there are underlying bugs here, but as we have been just retrying the failures and shipping code anyway, I don't think these tests are really adding any value. We should fix these issues ASAP though because I believe this is mostly new flakiness introduced in the past few months.

[github-actions]

✅ Gradle check result for cf42dbd: SUCCESS

[Pallavi-AWS]

@sgup432 @kiranprakash154 @getsaurabh02 Just FYI. It seems likely there are underlying bugs here, but as we have been just retrying the failures and shipping code anyway, I don't think these tests are really adding any value. We should fix these issues ASAP though because I believe this is mostly new flakiness introduced in the past few months.

hi @sgup432 - can you please confirm if we need these testcases? If yes, how can we remove the flakiness as soon as possible? Have we established that the tiered caching race condition being worked on by @kiranprakash154 would resolve a subset of these testcases? thanks.

[kiranprakash154]

Hi @Pallavi-AWS and @andrross,
we do need these test cases, but they should never be flaky. The race condition I'm working on (#14032) is a separate issue and I dont think affects the flakiness of these tests.

I'll get to flaky tests right after resolving the race condition bug and have a fix soon.
(cc: @kkhatua @sgup432 )

[sgup432]

@Pallavi-AWS As @kiranprakash154 pointed out, we need these testcases. We are working on this separately to resolve all/most of these tests all at once.