forked from ad-freiburg/qlever
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add
DeltaTriplesManager (ad-freiburg#1603)
The already existing `DeltaTriples` class maintains a dynamically changing set of insertions and deletions relative to the original input data, together with a (single) local vocab. The class is not threadsafe and has to be used with care. In particular, concurrent update queries have to be serialized, and while a query makes use of the "delta triples", it has to be made sure that they are not changed over the course of the processing of that query. Both of these problems are solved by the new `DeltaTriplesManager` class. The index has a single object of this class. It maintains a single `DeltaTriples` object, write access to which is strictly serialized. Each new query gets a so-called *snapshot* of the current delta triples. This is a full copy (of the delta triples located in each of the permutations and of the local vocab). These snapshots are read-only and multiple queries can share the same snapshot. A snapshot lives as long as one query using it is still being processed.
- Loading branch information
Showing
22 changed files
with
497 additions
and
211 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,7 @@ | ||
| // Copyright 2011, University of Freiburg, | ||
| // Chair of Algorithms and Data Structures. | ||
| // Author: | ||
| // 2011-2017 Björn Buchhold ([email protected]) | ||
| // 2018- Johannes Kalmbach ([email protected]) | ||
| // Copyright 2011 - 2024, University of Freiburg | ||
| // Chair of Algorithms and Data Structures | ||
| // Authors: Björn Buchhold <[email protected]> [2011 - 2017] | ||
| // Johannes Kalmbach <[email protected]> [2017 - 2024] | ||
|
|
||
| #pragma once | ||
|
|
||
|
|
@@ -92,7 +91,10 @@ class QueryExecutionContext { | |
|
|
||
| [[nodiscard]] const Index& getIndex() const { return _index; } | ||
|
|
||
| const DeltaTriples& deltaTriples() const { return *deltaTriples_; } | ||
| const LocatedTriplesSnapshot& locatedTriplesSnapshot() const { | ||
| AD_CORRECTNESS_CHECK(sharedLocatedTriplesSnapshot_ != nullptr); | ||
| return *sharedLocatedTriplesSnapshot_; | ||
| } | ||
|
|
||
| void clearCacheUnpinnedOnly() { getQueryTreeCache().clearUnpinnedOnly(); } | ||
|
|
||
|
|
@@ -123,10 +125,13 @@ class QueryExecutionContext { | |
|
|
||
| private: | ||
| const Index& _index; | ||
| // TODO<joka921> This has to be stored externally once we properly support | ||
| // SPARQL UPDATE, currently it is just a stub to make the interface work. | ||
| std::shared_ptr<DeltaTriples> deltaTriples_{ | ||
| std::make_shared<DeltaTriples>(_index)}; | ||
|
|
||
| // When the `QueryExecutionContext` is constructed, get a stable read-only | ||
| // snapshot of the current (located) delta triples. These can then be used | ||
| // by the respective query without interfering with further incoming | ||
| // update operations. | ||
| SharedLocatedTriplesSnapshot sharedLocatedTriplesSnapshot_{ | ||
| _index.deltaTriplesManager().getCurrentSnapshot()}; | ||
| QueryResultCache* const _subtreeCache; | ||
| // allocators are copied but hold shared state | ||
| ad_utility::AllocatorWithLimit<Id> _allocator; | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,7 @@ | ||
| // Copyright 2015, University of Freiburg, | ||
| // Chair of Algorithms and Data Structures. | ||
| // Author: | ||
| // 2015-2017 Björn Buchhold ([email protected]) | ||
| // 2018- Johannes Kalmbach ([email protected]) | ||
| // Copyright 2015 - 2024, University of Freiburg | ||
| // Chair of Algorithms and Data Structures | ||
| // Authors: Björn Buchhold <[email protected]> [2015 - 2017] | ||
| // Johannes Kalmbach <[email protected]> [2017 - 2024] | ||
|
|
||
| #include "./QueryExecutionTree.h" | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,8 @@ | ||
| // Copyright 2015, University of Freiburg, | ||
| // Chair of Algorithms and Data Structures. | ||
| // Author: Björn Buchhold ([email protected]) | ||
| // Copyright 2015 - 2024, University of Freiburg | ||
| // Chair of Algorithms and Data Structures | ||
| // Authors: Björn Buchhold <[email protected]> | ||
| // Johannes Kalmbach <[email protected]> | ||
|
|
||
| #pragma once | ||
|
|
||
| #include <memory> | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,8 @@ | ||
| // Copyright 2023 - 2024, University of Freiburg | ||
| // Chair of Algorithms and Data Structures. | ||
| // Authors: | ||
| // 2023 Hannah Bast <bast@cs.uni-freiburg.de> | ||
| // 2024 Julian Mundhahs <mundhahj@tf.uni-freiburg.de> | ||
| // Chair of Algorithms and Data Structures | ||
| // Authors: Hannah Bast <[email protected]> | ||
| // Julian Mundhahs <mundhahj@tf.uni-freiburg.de> | ||
| // Johannes Kalmbach <kalmbach@cs.uni-freiburg.de> | ||
|
|
||
| #include "index/DeltaTriples.h" | ||
|
|
||
|
|
@@ -21,8 +21,7 @@ LocatedTriples::iterator& DeltaTriples::LocatedTripleHandles::forPermutation( | |
| void DeltaTriples::clear() { | ||
| triplesInserted_.clear(); | ||
| triplesDeleted_.clear(); | ||
| std::ranges::for_each(locatedTriplesPerBlock_, | ||
| &LocatedTriplesPerBlock::clear); | ||
| std::ranges::for_each(locatedTriples(), &LocatedTriplesPerBlock::clear); | ||
| } | ||
|
|
||
| // ____________________________________________________________________________ | ||
|
|
@@ -33,15 +32,15 @@ DeltaTriples::locateAndAddTriples(CancellationHandle cancellationHandle, | |
| std::array<std::vector<LocatedTriples::iterator>, Permutation::ALL.size()> | ||
| intermediateHandles; | ||
| for (auto permutation : Permutation::ALL) { | ||
| auto& perm = index_.getImpl().getPermutation(permutation); | ||
| auto& perm = index_.getPermutation(permutation); | ||
| auto locatedTriples = LocatedTriple::locateTriplesInPermutation( | ||
| // TODO<qup42>: replace with `getAugmentedMetadata` once integration | ||
| // is done | ||
| idTriples, perm.metaData().blockData(), perm.keyOrder(), shouldExist, | ||
| cancellationHandle); | ||
| cancellationHandle->throwIfCancelled(); | ||
| intermediateHandles[static_cast<size_t>(permutation)] = | ||
| locatedTriplesPerBlock_[static_cast<size_t>(permutation)].add( | ||
| this->locatedTriples()[static_cast<size_t>(permutation)].add( | ||
| locatedTriples); | ||
| cancellationHandle->throwIfCancelled(); | ||
| } | ||
|
|
@@ -60,8 +59,8 @@ void DeltaTriples::eraseTripleInAllPermutations(LocatedTripleHandles& handles) { | |
| // Erase for all permutations. | ||
| for (auto permutation : Permutation::ALL) { | ||
| auto ltIter = handles.forPermutation(permutation); | ||
| locatedTriplesPerBlock_[static_cast<int>(permutation)].erase( | ||
| ltIter->blockIndex_, ltIter); | ||
| locatedTriples()[static_cast<int>(permutation)].erase(ltIter->blockIndex_, | ||
| ltIter); | ||
| } | ||
| } | ||
|
|
||
|
|
@@ -172,7 +171,48 @@ void DeltaTriples::modifyTriplesImpl(CancellationHandle cancellationHandle, | |
| } | ||
|
|
||
| // ____________________________________________________________________________ | ||
| const LocatedTriplesPerBlock& DeltaTriples::getLocatedTriplesPerBlock( | ||
| const LocatedTriplesPerBlock& | ||
| LocatedTriplesSnapshot::getLocatedTriplesForPermutation( | ||
| Permutation::Enum permutation) const { | ||
| return locatedTriplesPerBlock_[static_cast<int>(permutation)]; | ||
| } | ||
|
|
||
| // ____________________________________________________________________________ | ||
| SharedLocatedTriplesSnapshot DeltaTriples::getSnapshot() const { | ||
| // NOTE: Both members of the `LocatedTriplesSnapshot` are copied, but the | ||
| // `localVocab_` has no copy constructor (in order to avoid accidental | ||
| // copies), hence the explicit `clone`. | ||
| return SharedLocatedTriplesSnapshot{std::make_shared<LocatedTriplesSnapshot>( | ||
| locatedTriples(), localVocab_.clone())}; | ||
| } | ||
|
|
||
| // ____________________________________________________________________________ | ||
| DeltaTriples::DeltaTriples(const Index& index) | ||
| : DeltaTriples(index.getImpl()) {} | ||
|
|
||
| // ____________________________________________________________________________ | ||
| DeltaTriplesManager::DeltaTriplesManager(const IndexImpl& index) | ||
| : deltaTriples_{index}, | ||
| currentLocatedTriplesSnapshot_{deltaTriples_.rlock()->getSnapshot()} {} | ||
|
|
||
| // _____________________________________________________________________________ | ||
| void DeltaTriplesManager::modify( | ||
| const std::function<void(DeltaTriples&)>& function) { | ||
| // While holding the lock for the underlying `DeltaTriples`, perform the | ||
| // actual `function` (typically some combination of insert and delete | ||
| // operations) and (while still holding the lock) update the | ||
| // `currentLocatedTriplesSnapshot_`. | ||
| deltaTriples_.withWriteLock([this, &function](DeltaTriples& deltaTriples) { | ||
| function(deltaTriples); | ||
| auto newSnapshot = deltaTriples.getSnapshot(); | ||
| currentLocatedTriplesSnapshot_.withWriteLock( | ||
| [&newSnapshot](auto& currentSnapshot) { | ||
| currentSnapshot = std::move(newSnapshot); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| // _____________________________________________________________________________ | ||
| SharedLocatedTriplesSnapshot DeltaTriplesManager::getCurrentSnapshot() const { | ||
| return *currentLocatedTriplesSnapshot_.rlock(); | ||
| } | ||
Oops, something went wrong.