This repository has been archived by the owner on Sep 27, 2019. It is now read-only.

[15721] TileGroup Compaction and GC Fixes#1349

Open

dqrs wants to merge 121 commits intocmu-db:masterfrom mbutrovich:gc_fixes

+3,433-625

Commits on Jun 18, 2018

Created GC CommitDelete Test. Made ClearGarbage public in TLGC

David Gershuni
authored and
Matt Butrovich
committed
Modified CommitDelete test so that it properly cleans up the database

David Gershuni
authored and
Matt Butrovich
committed
Added 14 tests to transaction-level GC manager. Captures 4 GC failures.

David Gershuni
authored and
Matt Butrovich
committed
Added test utilities, and multiple new test cases for checking

David Gershuni
authored and
Matt Butrovich
committed
Added more index tests. Added tests for primary key updates.

David Gershuni
authored and
Matt Butrovich
committed
Added PrimaryKeyUpdateTest

mbutrovich
authored and
Matt Butrovich
committed
Refactor.

mbutrovich
authored and
Matt Butrovich
committed
Fixed eror in CommitUpdateSecondaryKeyTest.

mbutrovich
authored and
Matt Butrovich
committed
Fixed bug where tuple slots are not reclaimed when insertions fail.

David Gershuni
authored and
Matt Butrovich
committed
Updated transaction manager and garbage collector to properly handle

David Gershuni
authored and
Matt Butrovich
committed
Updated GetRecycledTupleSlot() in GC to no longer hand out slots from

David Gershuni
authored and
Matt Butrovich
committed
Enhanced GC tests to check indexes in all cases, uses new test function

David Gershuni
authored and
Matt Butrovich
committed
Modified the GC so that it properly removes index entries for

David Gershuni
authored and
Matt Butrovich
committed
Minor refactor and comments before a formatting run.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 on modified files.

mbutrovich
authored and
Matt Butrovich
committed
Created GC CommitDelete Test. Made ClearGarbage public in TLGC

David Gershuni
authored and
Matt Butrovich
committed
Modified CommitDelete test so that it properly cleans up the database

David Gershuni
authored and
Matt Butrovich
committed
Added 14 tests to transaction-level GC manager. Captures 4 GC failures.

David Gershuni
authored and
Matt Butrovich
committed
Added test utilities, and multiple new test cases for checking

David Gershuni
authored and
Matt Butrovich
committed
Added more index tests. Added tests for primary key updates.

David Gershuni
authored and
Matt Butrovich
committed
Added PrimaryKeyUpdateTest

mbutrovich
authored and
Matt Butrovich
committed
Refactor.

mbutrovich
authored and
Matt Butrovich
committed
Fixed bug where tuple slots are not reclaimed when insertions fail.

David Gershuni
authored and
Matt Butrovich
committed
Updated transaction manager and garbage collector to properly handle

David Gershuni
authored and
Matt Butrovich
committed
Enhanced GC tests to check indexes in all cases, uses new test function

David Gershuni
authored and
Matt Butrovich
committed
Modified the GC so that it properly removes index entries for

David Gershuni
authored and
Matt Butrovich
committed
Minor refactor and comments before a formatting run.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 on modified files.

mbutrovich
authored and
Matt Butrovich
committed
Disabled 4 failing tests that are not addressed in this PR and will open a new issue for those after this is merged.

mbutrovich
authored and
Matt Butrovich
committed
Reenabled 4 disabled tests because we still want to test recycle slots. Disabled the index checks in those tests until that issue is resolved.

mbutrovich
authored and
Matt Butrovich
committed
Fixed unused attribute

poojanilangekar
authored and
Matt Butrovich
committed
Revert "clang-format-3.6 on modified files."

poojanilangekar
authored and
Matt Butrovich
committed
Revert "clang-format-3.6 again after rebase."

poojanilangekar
authored and
Matt Butrovich
committed
Revert some remaining format changes + disable tests

poojanilangekar
authored and
Matt Butrovich
committed
Applied all TileGroup compaction changes to the GC Fixes branch.

David Gershuni
authored and
Matt Butrovich
committed
Created GC CommitDelete Test. Made ClearGarbage public in TLGC

David Gershuni
authored and
Matt Butrovich
committed
Modified CommitDelete test so that it properly cleans up the database

David Gershuni
authored and
Matt Butrovich
committed
Added 14 tests to transaction-level GC manager. Captures 4 GC failures.

David Gershuni
authored and
Matt Butrovich
committed
Added test utilities, and multiple new test cases for checking

David Gershuni
authored and
Matt Butrovich
committed
Added more index tests. Added tests for primary key updates.

David Gershuni
authored and
Matt Butrovich
committed
Added PrimaryKeyUpdateTest

mbutrovich
authored and
Matt Butrovich
committed
Refactor.

mbutrovich
authored and
Matt Butrovich
committed
Fixed bug where tuple slots are not reclaimed when insertions fail.

David Gershuni
authored and
Matt Butrovich
committed
Enhanced GC tests to check indexes in all cases, uses new test function

David Gershuni
authored and
Matt Butrovich
committed
Minor refactor and comments before a formatting run.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 on modified files.

mbutrovich
authored and
Matt Butrovich
committed
Created GC CommitDelete Test. Made ClearGarbage public in TLGC

David Gershuni
authored and
Matt Butrovich
committed
Modified CommitDelete test so that it properly cleans up the database

David Gershuni
authored and
Matt Butrovich
committed
Added 14 tests to transaction-level GC manager. Captures 4 GC failures.

David Gershuni
authored and
Matt Butrovich
committed
Added test utilities, and multiple new test cases for checking

David Gershuni
authored and
Matt Butrovich
committed
Added more index tests. Added tests for primary key updates.

David Gershuni
authored and
Matt Butrovich
committed
Added PrimaryKeyUpdateTest

mbutrovich
authored and
Matt Butrovich
committed
Refactor.

mbutrovich
authored and
Matt Butrovich
committed
Fixed eror in CommitUpdateSecondaryKeyTest.

mbutrovich
authored and
Matt Butrovich
committed
Fixed bug where tuple slots are not reclaimed when insertions fail.

David Gershuni
authored and
Matt Butrovich
committed
Updated transaction manager and garbage collector to properly handle

David Gershuni
authored and
Matt Butrovich
committed
Updated GetRecycledTupleSlot() in GC to no longer hand out slots from

David Gershuni
authored and
Matt Butrovich
committed
Enhanced GC tests to check indexes in all cases, uses new test function

David Gershuni
authored and
Matt Butrovich
committed
Minor refactor and comments before a formatting run.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 on modified files.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 again after rebase.

mbutrovich
authored and
Matt Butrovich
committed
Disabled 4 failing tests that are not addressed in this PR and will open a new issue for those after this is merged.

mbutrovich
authored and
Matt Butrovich
committed
Reenabled 4 disabled tests because we still want to test recycle slots. Disabled the index checks in those tests until that issue is resolved.

mbutrovich
authored and
Matt Butrovich
committed
Revert "clang-format-3.6 on modified files."

poojanilangekar
authored and
Matt Butrovich
committed
Revert "clang-format-3.6 again after rebase."

poojanilangekar
authored and
Matt Butrovich
committed
Revert some remaining format changes + disable tests

poojanilangekar
authored and
Matt Butrovich
committed
Applied all TileGroup compaction changes to the GC Fixes branch.

David Gershuni
authored and
Matt Butrovich
committed
Minor fixes after rebase.

mbutrovich
authored and
Matt Butrovich
committed
More minor fixes after rebase.

mbutrovich
authored and
Matt Butrovich
committed
Fixed table naming to be lowercase to fix exception in table generation.

mbutrovich
authored and
Matt Butrovich
committed
Fixed errant copy-paste.

mbutrovich
authored and
Matt Butrovich
committed
Removed reliance on GC Manager's tables_ structure. Go to the storage manager instead. This is for PR#2 for 15721.

mbutrovich
authored and
Matt Butrovich
committed
Changed tile group freeing approach. Going to use lock-free stack instead.

David Gershuni
authored and
Matt Butrovich
committed
Finished initial draft of GCManager. Renamed and refactored UnlinkVersion(s)

David Gershuni
authored and
Matt Butrovich
committed
Created RecycleStack class. Not yet tested.

David Gershuni
authored and
Matt Butrovich
committed
Bug fixes, compiles, runs transaction_level_gc_manager_tests.

mbutrovich
authored and
Matt Butrovich
committed
Minor bug fixes. Passes make check. Need to write more tests to execrise new GCManager.

mbutrovich
authored and
Matt Butrovich
committed
Added a way to lookup elements in LockFreeArray. Need this to do a reverse-mapping from TileGroup oid to offset. Reenabled DropTileGroupsTest. Passes make check.

mbutrovich
authored and
Matt Butrovich
committed
Preliminary implementation of TileGroup compaction. Needs to be tested.

David Gershuni
authored and
Matt Butrovich
committed
Bug fix in compaction (removed LogicalTile), and added a new test for tile_group_compactor.

mbutrovich
authored and
Matt Butrovich
committed
Made immutable_tile_group_queue a single queue instead of multiple, made access function to add to it, and made SetImmutability notify the GC.

mbutrovich
authored and
Matt Butrovich
committed
Moved TileGroup compaction to a stand-alone class, separate from the GC.

mbutrovich
authored and
Matt Butrovich
committed
Modifications to handle removing TileGroups: made manager.h actually remove elements instead of a sentinel value on removal, and got rid of TileGroupIterator.

mbutrovich
authored and
Matt Butrovich
committed
2 small changes based on PR comments.

mbutrovich
authored and
Matt Butrovich
committed
Refactor to Manager class for TileGroup dropping.

mbutrovich
authored and
Matt Butrovich
committed
Fixed typo in manager.cpp.

mbutrovich
authored and
Matt Butrovich
committed
Refactor and comments regarding immutability.

mbutrovich
authored and
Matt Butrovich
committed
Reenabled old GC tests. Modified immutability test to be correct with recycled tombstones. It previously assumed no tombstones were recycled.

mbutrovich
authored and
Matt Butrovich
committed
Refactor of TransactionLevelGCManager. Added compaction queue.

David Gershuni
authored and
Matt Butrovich
committed
Audited codebase to find all places where tile_groups dereferenced

David Gershuni
authored and
Matt Butrovich
committed
Created TileGroupCompactor concurrency test.

David Gershuni
authored and
Matt Butrovich
committed
Added several TileGroupCompactor tests

David Gershuni
authored and
Matt Butrovich
committed
Fixed two possible nullptr dereferences in TOTM, added some LOG_TRACE to help debugging.

mbutrovich
authored and
Matt Butrovich
committed
Final pass of comments. Added Setter for GC.compaction_threshold_.

David Gershuni
authored and
Matt Butrovich
committed
Added global setting for TileGroup freeing, default is off.

mbutrovich
authored and
Matt Butrovich
committed
RecycleStack Doxygen comments.

mbutrovich
authored and
Matt Butrovich
committed
TileGroupCompactor Doxygen comments.

mbutrovich
authored and
Matt Butrovich
committed
TransactionLevelGCManager Doxygen comments.

mbutrovich
authored and
Matt Butrovich
committed
Minor refactor and clang-tidy changes.

mbutrovich
authored and
Matt Butrovich
committed
More minor refactors and changes based on PR feedback.

mbutrovich
authored and
Matt Butrovich
committed
Added a boolean to disable compaction entirely.

mbutrovich
authored and
Matt Butrovich
committed
Putting back the TileGroupIterator for future enhancement.

mbutrovich
authored and
Matt Butrovich
committed
Fixed TRACE-level logging issues in GC manager, added some Doxygen comments to TileGroupHeader for new/modified functions.

mbutrovich
authored and
Matt Butrovich
committed
Post-rebase fixes.

mbutrovich
authored and
Matt Butrovich
committed
Fixed bug in manager.cpp after rebase. Removed unnecessary variables in tile_group_compactor_test, fixed bug in RetryTest.

mbutrovich
authored and
Matt Butrovich
committed
clang-format-3.6 on modified lines.

mbutrovich
authored and
Matt Butrovich
committed
Fix formatting issue on recycle_stack's include.

mbutrovich
authored and
Matt Butrovich
committed
Fix unused variables when tests compiled in Release mode.

mbutrovich
authored and
Matt Butrovich
committed
Fix printing issue in tile_group_compactor_test and reduced LOG_DEBUGs in GCManager.

mbutrovich
authored and
Matt Butrovich
committed
Fix unused variables in Release mode.

mbutrovich
authored and
Matt Butrovich
committed
Fix Debug printing issue that last commit introduced.

mbutrovich
authored and
Matt Butrovich
committed
Added IsRunning member to MonoQueuePool so TileGroupCompactor can check if system is shutting down to avoid a race and stop trying to compact. Resolves segfault in tile_group_compactor_test.

mbutrovich
authored and
Matt Butrovich
committed
Fix include issue for GCC on Travis.

mbutrovich
authored and
Matt Butrovich
committed
Fix memory ordering issue for GCC on Travis.

mbutrovich
authored and
Matt Butrovich
committed
Added more comments to disabled tests, fixed typo in tests.

mbutrovich
authored and
Matt Butrovich
committed
New test scenario that triggers an assertion fail in TOTM (CommitInsertUpdatePrimaryKetTest).

mbutrovich
authored and
Matt Butrovich
committed
Post-rebase fixes.

mbutrovich
authored and
Matt Butrovich
committed
More post-rebase fixes. Passes make check again.

mbutrovich
authored and
Matt Butrovich
committed
Fix condition in tile_group_compactor test that was too strict for some systems due to performance differences.

mbutrovich
authored and
Matt Butrovich
committed
Minor refactor and comment changes.

mbutrovich
authored and
Matt Butrovich
committed
Fix tile_group_compactor not recording ReadOwn behavior.
mbutrovich
committed