Optimize Aborted Scavenge

[amicic]

When aborted Scavenge occurs (in a middle of Scavnge, no free space left in either Nursery or Tenure to move objects to) we handle it differently for:

1. STW Scavenge, backout the work that Scavenge performed in this cycle (move the objects and references to their original position and values), and after that perform an ordinary Global GC to free up memory in Nursery by sweep and proceed with allocation (more aggressive Global GCs may be triggered if needed)

2. CS Scavenge, we stop, leave the objects where they are, and perform Global GC that is aware of forwarding pointers. It fixes up all the references that have not been updated. While sweep will also free up some memory in Nursery, it will be more fragmented (since both Evacuate and Survivor side of Nursery will have live objects scattered), a single threaded compaction is performed to increase the likelihood of allocation success.

The latter approach could work for both CS and STW (and the former does not work for CS), so we could simplify the code and only use the latter one.

Another motivation for not doing backout approach, is that we could remove the second pass of RS scanning (aka pruning pass), hence slightly reduce Scavenge time (a rough estimate, by 5%).

However doing a single thread compaction is very expensive and should be avoided and optimized if/when possible:

• we can try Global GC without compact, and only if that allocation fails, we trigger compaction (or simply another Global GC with compaction)
• we try single thread Compact only if parallel Compact does not satisfy allocation
• we can Compact only Nursery (if the original Scavenge was triggered by Nursery allocation) - we can do this because we have correct RS after Global GC. We would fixup RS, but not need to Compact Tenure, which typically has far more objects.

[github-actions]

Issue Number: 20515
Status: Open
Recommended Components: comp:gc, comp:vm, comp:test
Recommended Assignees: amicic, tajila, babsingh