Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices #13453

Merged
merged 2 commits into from
Apr 30, 2024

Conversation

kiranprakash154
Copy link
Contributor

@kiranprakash154 kiranprakash154 commented Apr 29, 2024

Description

Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices. failing with java.lang.AssertionError: expected:<-665> but was:<665>

In the below code, memorySizeOfLatestEntryForIndex1 would be negative (-665) if the cacheCleaner ran before the execution of that line resulting in getRequestCacheStats(client, index1).getMemorySizeInBytes() to be smaller than finalMemorySizeForIndex1

long memorySizeOfLatestEntryForIndex1 = getRequestCacheStats(client, index1).getMemorySizeInBytes() - finalMemorySizeForIndex1;
.
.
.
assertEquals(memorySizeOfLatestEntryForIndex1, getRequestCacheStats(client, index1).getMemorySizeInBytes())

What is the Fix ?

The fix is to not depend on the exact size of the memorySizeOfLatestEntryForIndex1 and instead make sure the current request cache stats size is less than memorySizeForIndex1With2Entries and > 0
By this, we avoid the race condition of cacheCleaner being invoked before the calculation of memorySizeOfLatestEntryForIndex1

Related Issues

Resolves #13437

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@kiranprakash154 kiranprakash154 added flaky-test Random test failure that succeeds on second run skip-changelog labels Apr 29, 2024
@kiranprakash154 kiranprakash154 self-assigned this Apr 29, 2024
@kiranprakash154 kiranprakash154 changed the title Update IndicesRequestCacheIT.java Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices Apr 29, 2024
@github-actions github-actions bot added bug Something isn't working Search Search query, autocomplete ...etc labels Apr 29, 2024
Copy link
Contributor

❌ Gradle check result for d81d42a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@kiranprakash154
Copy link
Contributor Author

❌ Gradle check result for d81d42a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Tests with failures:

Copy link
Contributor

✅ Gradle check result for d81d42a: SUCCESS

Copy link

codecov bot commented Apr 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.46%. Comparing base (b15cb0c) to head (5b6926e).
Report is 239 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13453      +/-   ##
============================================
+ Coverage     71.42%   71.46%   +0.04%     
- Complexity    59978    60839     +861     
============================================
  Files          4985     5046      +61     
  Lines        282275   286108    +3833     
  Branches      40946    41432     +486     
============================================
+ Hits         201603   204470    +2867     
- Misses        63999    64786     +787     
- Partials      16673    16852     +179     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

❕ Gradle check result for 5b6926e: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@kiranprakash154
Copy link
Contributor Author

❕ Gradle check result for 5b6926e: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Known Flaky #11547

@kiranprakash154
Copy link
Contributor Author

kiranprakash154 commented Apr 29, 2024

@dblock can u merge this ?
(cc: @kkhatua )

@kiranprakash154 kiranprakash154 added backport 2.x Backport to 2.x branch backport 2.0 Backport to 2.0 branch and removed backport 2.0 Backport to 2.0 branch labels Apr 29, 2024
@dblock dblock merged commit 51f8872 into opensearch-project:main Apr 30, 2024
37 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 30, 2024
…Indices (#13453)

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>
(cherry picked from commit 51f8872)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gbbafna pushed a commit that referenced this pull request Apr 30, 2024
…Indices (#13453) (#13459)

(cherry picked from commit 51f8872)

Signed-off-by: Kiran Prakash <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
finnegancarroll pushed a commit to finnegancarroll/OpenSearch that referenced this pull request May 10, 2024
…Indices (opensearch-project#13453)

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>
@dblock
Copy link
Member

dblock commented May 15, 2024

Happened again in #13655.

deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request May 17, 2024
…Indices (opensearch-project#13453)

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Sep 3, 2024
…Indices (opensearch-project#13453)

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update IndicesRequestCacheIT.java

Signed-off-by: Kiran Prakash <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Sep 4, 2024
commit 93a3132
Author: Peter Alfonsi <[email protected]>
Date:   Wed Sep 4 11:44:03 2024 -0700

    fix broken ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

commit fd92c5a
Author: Marc Handalian <[email protected]>
Date:   Thu Jan 11 06:41:55 2024 -0800

    Update runTask to optionally install plugins (opensearch-project#11844)

    Signed-off-by: Marc Handalian <[email protected]>

commit 4d78661
Author: Peter Alfonsi <[email protected]>
Date:   Tue Sep 3 16:02:31 2024 -0700

    Misc test fixes

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 70a6935
Author: Peter Alfonsi <[email protected]>
Date:   Fri Aug 23 10:42:05 2024 -0700

    [Bugfix] Fixes IRC NPE bug for timed-out cacheable queries (opensearch-project#15327)

    * Fix IRC timeout bug

    Signed-off-by: Peter Alfonsi <[email protected]>

    * addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Add UT for test coverage

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * tweak imports in new UT

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit f9943f8
Author: Sagar <[email protected]>
Date:   Wed Jul 17 19:02:05 2024 -0700

    Clear ehcache disk cache files during initialization (opensearch-project#14738)

    * Clear ehcache disk cache files during initialization

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding UT to fix line coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding more Uts for better line coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Throwing exception in case we fail to clear cache files during startup

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding more UTs

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a UT for more coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle build

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Update ehcache disk cache close() logic

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit fc2007e
Author: Peter Alfonsi <[email protected]>
Date:   Mon Jul 1 15:54:39 2024 -0700

    [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564)

    * Fix testInvalidInput flakiness

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun security check

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 1ef62e2
Author: Sagar <[email protected]>
Date:   Wed Jun 26 11:14:42 2024 -0700

    Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550)

    * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 29ec9d6
Author: Sagar <[email protected]>
Date:   Tue Jun 25 12:04:17 2024 -0700

    [Tiered Caching]  Moving query recomputation logic outside of write lock (opensearch-project#14187)

    * Moving query recompute out of write lock

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * [Tiered Caching] Moving query recomputation logic outside of write lock

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding java doc for the completable map

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Changes to call future handler only once per key

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing spotless check

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle fail

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments to refactor unit test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * minor UT refactor

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 8347779
Author: Peter Alfonsi <[email protected]>
Date:   Tue Jun 25 13:26:54 2024 -0700

    Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462)

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 40f868d
Author: Kiran Prakash <[email protected]>
Date:   Thu Jun 20 14:17:23 2024 -0700

    Update IndicesRequestCacheCleanupIT.java (opensearch-project#14478)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 094a9f2
Author: Kiran Prakash <[email protected]>
Date:   Wed Jun 19 15:36:59 2024 -0700

    Fix Indices Request Cache ITs from flaking and timing out (opensearch-project#14369)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 3a110a8
Author: Kiran Prakash <[email protected]>
Date:   Tue Jun 18 10:18:25 2024 -0700

    [Tiered Cache] Use ConcurrentHashMap explicitly in IndicesRequestCache (opensearch-project#14409)

    Signed-off-by: Kiran Prakash <[email protected]>

commit bcf0f59
Author: Peter Alfonsi <[email protected]>
Date:   Fri Jun 14 13:20:51 2024 -0700

    [Bugfix] Fix TieredSpilloverCache flaky tests (opensearch-project#14333)

    * Fix flaky TSC stats tests

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <[email protected]>

    * fix forbidden API

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit a222a03
Author: Kiran Prakash <[email protected]>
Date:   Wed Jun 12 18:30:06 2024 -0700

    [Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception (opensearch-project#14221)

    * use concurrentmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert feature flags

    Signed-off-by: Kiran Prakash <[email protected]>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <[email protected]>

    * use concurrentmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert feature flags

    Signed-off-by: Kiran Prakash <[email protected]>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert the test removal

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert the conflict resolutions

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f85cd00
Author: Sagar <[email protected]>
Date:   Wed Jun 12 08:22:38 2024 -0700

    Fix ShardNotFoundException during request cache clean up (opensearch-project#14219)

    * Fix for ShardNotFoundException during request cache clean up

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fix forbidden gradle check

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 406ae78
Author: Sagar <[email protected]>
Date:   Thu Jun 6 15:02:00 2024 -0700

    [Caching] Move cache removal notifications outside lru lock (opensearch-project#14017)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit c5fd7fe
Author: Peter Alfonsi <[email protected]>
Date:   Tue Sep 3 14:38:26 2024 -0700

    Change IRC UT to correctly create shard

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 9c4d4af
Author: Peter Alfonsi <[email protected]>
Date:   Tue Jun 4 16:09:18 2024 -0700

    [Tiered Caching] Additional ITs for cache stats  (opensearch-project#13655)

    * Adds cache clear IT

    Signed-off-by: Peter Alfonsi <[email protected]>

    Cleaned up logic for cache stats ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

    Adds more tests around tiered spillover cache

    Signed-off-by: Peter Alfonsi <[email protected]>

    Fixed cache stats behavior for overall /_nodes/stats call

    Signed-off-by: Peter Alfonsi <[email protected]>

    cleanup

    Signed-off-by: Peter Alfonsi <[email protected]>

    Fixed folder structure

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    Break horrifyingly long test case into many shorter cases

    Signed-off-by: Peter Alfonsi <[email protected]>

    Added unsupported operation exception to TSC stats holder incrementEvictions()

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun assemble

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit be97e90
Author: Peter Alfonsi <[email protected]>
Date:   Thu May 23 12:02:22 2024 -0700

    [Bugfix] [Tiered Caching] Fixes issues when integrating tiered cache with disk cache (opensearch-project#13784)

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 60ee715
Author: Kiran Prakash <[email protected]>
Date:   Mon May 20 12:52:03 2024 -0700

    Update IndicesRequestCacheIT.java (opensearch-project#13678)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 51f6c1b
Author: Sagar <[email protected]>
Date:   Wed May 8 09:15:36 2024 -0700

    Fix IndicesRequestCache clean up logic (opensearch-project#13597)

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 49e2701
Author: Sagar <[email protected]>
Date:   Mon May 6 17:25:45 2024 -0700

    Fix negative requestStats memory_size issue (opensearch-project#13553)

    This solves the bug where RequestStats memory_size metric was going negative in certain scenarios
    as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then
    reallocated on the same node. So whenever stale entries from older shard are deleted, those are
    accounted for the new shard which has the same shardId.

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit a5c5675
Author: Peter Alfonsi <[email protected]>
Date:   Thu May 2 17:29:07 2024 -0700

    [Tiered Caching] Adds stats implementation for TieredSpilloverCache (opensearch-project#13236)

    Stats rework part 4 of 4
    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 46944cb
Author: peteralfonsi <[email protected]>
Date:   Tue Apr 30 10:59:03 2024 -0700

    Fix flaky test CacheStatsAPIIndicesRequestCacheIT.testNullLevels() (opensearch-project#13457)

    * Fix flaky test

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Initialize CommonStatsFlags with empty array for levels

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Fixes tests using incorrect null levels

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 71fcad4
Author: Kiran Prakash <[email protected]>
Date:   Mon Apr 29 18:41:33 2024 -0700

    Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices (opensearch-project#13453)

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f60339f
Author: peteralfonsi <[email protected]>
Date:   Mon Apr 29 22:37:05 2024 -0700

    [Tiered Caching] Bump versions for serialization in new cache stats API (opensearch-project#13460)

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 88718a2
Author: peteralfonsi <[email protected]>
Date:   Mon Apr 29 14:47:52 2024 -0700

    [Tiered Caching] Expose new cache stats API (opensearch-project#13237)

    Step 3 out of 4

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 5e97082
Author: peteralfonsi <[email protected]>
Date:   Sun Apr 28 08:15:18 2024 -0700

    [Tiered Caching] Gate CacheStatsHolder logic behind FeatureFlags.PLUGGABLE_CACHE setting (opensearch-project#13238)

    Stats rework step 2 of 4

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit ed881f7
Author: Kiran Prakash <[email protected]>
Date:   Fri Apr 26 11:52:31 2024 -0700

    [Tiered Caching] Make Indices Request Cache Stale Key Mgmt Threshold setting dynamic (opensearch-project#12941)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update ClusterSettings.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * some refactoring

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address existing tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * UTs

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * ITs

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * refactor

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * address code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * address code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit 424ccc9
Author: Sagar <[email protected]>
Date:   Thu Apr 25 22:09:42 2024 -0700

    [Tiered Caching] Expose a dynamic setting to disable/enable disk cache (opensearch-project#13373)

    * [Tiered Caching] Expose a dynamic setting to disable/enable disk cache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Putting tiered cache settings behind feature flag

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing Sorabh's comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Putting new setting behind feature flag

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit acbbb2f
Author: Kiran Prakash <[email protected]>
Date:   Wed Apr 24 14:50:39 2024 -0700

    [Tiered Caching] Bug fix for IndicesRequestCache StaleKey management (opensearch-project#13070)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * code comments only

    Signed-off-by: Kiran Prakash <[email protected]>

    * docs changes

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert catching AlreadyClosedException

    Signed-off-by: Kiran Prakash <[email protected]>

    * assert

    Signed-off-by: Kiran Prakash <[email protected]>

    * conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless apply

    Signed-off-by: Kiran Prakash <[email protected]>

    * address comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * update code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * address bug & add tests

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f1d2e72
Author: Sagar <[email protected]>
Date:   Mon Apr 15 10:43:22 2024 -0800

    [Tiered Caching] Ehcache Disk cache IT  (opensearch-project#12904)

    * Ehcache IT tests

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding some logs to print key/value size

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Add ehcache related invalidation IT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Remvoing unnecessary IndicesRequestCache IT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Indentation fix

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added tests around expiration time and invalidation

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit b0de406
Author: peteralfonsi <[email protected]>
Date:   Fri Apr 12 16:09:06 2024 -0700

    [Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers (opensearch-project#12531)

    As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains
    dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the
    CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by
    these dimensions.

    Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the
    TieredSpilloverCache will be in a followup PR.

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 40a8d34
Author: Sagar <[email protected]>
Date:   Thu Apr 11 14:49:03 2024 -0800

    [Tiered Caching] Make took time policy dynamic and add additional integ tests (opensearch-project#13063)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit 6f88777
Author: Sagar <[email protected]>
Date:   Tue Mar 19 19:41:59 2024 -0700

    Fixing ehcache flaky test (opensearch-project#12764)

    * Fixing ehcache flaky test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a ehcache issue reference for thread leak issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Updating comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit f69c089
Author: Sagar <[email protected]>
Date:   Mon Mar 18 22:20:52 2024 -0700

    [Tiered Caching] Clear up disk cache(ehcache) files during node shutdown (opensearch-project#12734)

    * Adding logic to clear up the disk cache files during close()
    * Adding logic to update entries count after invalidateAll()
    * Removing unneeded system log statement
    * Added comment in test for readability
    * Fixing issue where we were sending compacted byte[] array to ehcache but calculating size with padded byte[]

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit bf6488d
Author: Sagar <[email protected]>
Date:   Mon Mar 18 22:17:14 2024 -0700

    [Tiered Caching] Fix test testComputeIfAbsentWithFactoryBasedCacheCreation (opensearch-project#12700)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 0fc1f40
Author: peteralfonsi <[email protected]>
Date:   Mon Mar 18 13:39:35 2024 -0700

    [Tiered Caching] Serializers for ehcache (opensearch-project#12709)

    Adds serializers and integrates them into ehcache disk cache

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit a45be0f
Author: Kiran Prakash <[email protected]>
Date:   Sat Mar 16 17:12:27 2024 -0700

    [Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625)

    * Introduce IndicesRequestCacheCleanupManager

    Signed-off-by: Kiran Prakash <[email protected]>

    * using cleanup mgr to enqueue cleanups

    Signed-off-by: Kiran Prakash <[email protected]>

    * readability improvements

    Signed-off-by: Kiran Prakash <[email protected]>

    * update updateCleanupKeyToCountMap on new cache entry

    Signed-off-by: Kiran Prakash <[email protected]>

    * create IndicesRequestCacheCleanupManager & settings and validators

    Signed-off-by: Kiran Prakash <[email protected]>

    * Add IRC specific cache cleaner and remove from IndicesService

    Signed-off-by: Kiran Prakash <[email protected]>

    * Unit Tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * move cachecleaner inside mgr

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove processCleanupKeys

    Signed-off-by: Kiran Prakash <[email protected]>

    * minor cleanups

    Signed-off-by: Kiran Prakash <[email protected]>

    * add updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove locks and make all methods synchronized

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * Testing

    Signed-off-by: Kiran Prakash <[email protected]>

    * add Reschedule back to indices service

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename updateStaleKeysCount to incrementStaleKeysCount

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename getStaleKeysCountForTesting to getStaleKeysCount

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename threshold to stalenessThreshold

    Signed-off-by: Kiran Prakash <[email protected]>

    * check for cleanupKey.entity == null

    Signed-off-by: Kiran Prakash <[email protected]>

    * use computeIfPresent with keycountmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * log both staleKeysInCache & Staleness in debug logs

    Signed-off-by: Kiran Prakash <[email protected]>

    * Use HashMap instead of ConcurrentMap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Address b/w compatibility

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * make cleanCache synchronized

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove shouldRemoveKey

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotlessApply

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit bad412c
Author: Sagar <[email protected]>
Date:   Fri Mar 15 16:53:54 2024 -0700

    [Tiered caching] Supporting removal function on EhcacheDiskCache iterator (opensearch-project#12653)

    * [Tiered caching] Supporting removal function on EhcacheDiskCache iterator

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Minor refactoring in unit test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 15c4afa
Author: peteralfonsi <[email protected]>
Date:   Fri Mar 15 15:36:22 2024 -0700

    [Tiered Caching] Cache tier policies (opensearch-project#12542)

    * Adds policy interface and took time policy impl

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Changes IndicesService to write a CachePolicyInfoWrapper before the QSR

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Moved took time logic from QSR to IndicesService

    Signed-off-by: Peter Alfonsi <[email protected]>

    * spotlessApply

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed ansjcy's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Partial rebase on most recent changes

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Integrated policies with new TSC changes

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Reverted unintended change to idea/vcs.xml

    Signed-off-by: Peter Alfonsi <[email protected]>

    * javadocs

    Signed-off-by: Peter Alfonsi <[email protected]>

    * github actions

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Set default threshold value to 10 ms

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's second round of comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Set cachedQueryParser in IRC

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's comments besides dynamic setting

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Removed dynamic setting, misc comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added changelog entry

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added missing javadoc

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Fixed failed gradle run

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added setting validation test

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle for flaky IT

    Signed-off-by: Peter Alfonsi <[email protected]>

    * javadocs

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 45fce15
Author: Sagar <[email protected]>
Date:   Thu Mar 14 13:00:24 2024 -0700

    [Tiered Caching] Fixing flaky tiered cache test (opensearch-project#12650)

    * Fixing flaky tiered cache test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unnecessary comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unused variable

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit bb66699
Author: Peter Alfonsi <[email protected]>
Date:   Fri Aug 30 11:36:05 2024 -0700

    Attempt to fix broken ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 58170bf
Author: Sagar <[email protected]>
Date:   Mon Mar 11 14:12:55 2024 -0700

    [Tiered caching] Integrating IndicesRequestCache with CacheService controlled by a feature flag (opensearch-project#12533)

    * Adding changelog
    * Fixing gradle build issue
    * Fixing CacheService test
    * Adding UT in IndicesRequestCache with feature flag for more coverage
    * Updating changelog and renaming feature flag setting
    * Moving feature flag setting handling logic to CacheService by maintaining backward compatibility
    * Fixing broken UTs

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit 80d2130
Author: Sagar <[email protected]>
Date:   Fri Mar 1 11:10:28 2024 -0800

    [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option (opensearch-project#11874)

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit f372c2c
Author: Sagar <[email protected]>
Date:   Wed Jan 10 17:06:29 2024 -0800

    [Tiered Caching] Enable serialization of IndicesRequestCache.Key (opensearch-project#10275)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>
    Co-authored-by: Kiran Prakash <[email protected]>

commit b880c0b
Author: Sagar <[email protected]>
Date:   Tue Jan 9 10:57:52 2024 -0800

    [Tiered caching] Framework changes (opensearch-project#10753)

    * [Tiered caching] Framework changes

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added javadoc for new files/packages

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing javadoc warnings

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing additional minor comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Moving non null check to builder for OS onHeapCache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding package-info for new packages

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing service and adding different cache interfaces along with event listener support

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle missingDoc issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Changing listener logic, removing tiered cache integration with IRC

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding opensearch.internal tag for LoadAwareCacheLoader

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing thread safety issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Remove compute function and event listener logic change for TieredCache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Making Cache.compute function private

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding javadoc and more test for cache.put

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding write locks to refresh API as well

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unwanted EventType class and refactoring one UT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing TieredCache interface

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

Signed-off-by: Peter Alfonsi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch bug Something isn't working flaky-test Random test failure that succeeds on second run Search Search query, autocomplete ...etc skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Flaky test - org.opensearch.indices.IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices
3 participants