diff --git a/mini-lsm/src/test_your_understanding/Week1/Day1.md b/mini-lsm/src/test_your_understanding/Week1/Day1.md new file mode 100644 index 00000000..b97c3b5a --- /dev/null +++ b/mini-lsm/src/test_your_understanding/Week1/Day1.md @@ -0,0 +1,28 @@ +### Test Your Understanding + +- Why doesn't the memtable provide a delete API? + - Because we don't delete from LSM-backed storage in the hot path. We add a "tombstone" entry, and deletion is handled + later by the compaction process. Memtable simply stores a deleted key as a tombstone entry (either through empty value or explicit markers) + +- Is it possible to use other data structures as the memtable in LSM? What are the pros/cons of using the skiplist? + - We can use B-Tree or AVL tree as well. SkipLists are easier to implement and have good concurrency support. They take higher memory + and the balancing is probabilistic rather than guaranteed. + +- Why do we need a combination of state and state_lock? Can we only use state.read() and state.write()? + - `state_lock` helps take locks for operations that don't necessasrily involve get/put but other operations. This helps avoid + contention on the actual client workload while we are say "freezing a memtable". + +- Why does the order to store and to probe the memtables matter? If a key appears in multiple memtables, which version should you return to the user? + - The order to store and probe memtables is directly related to the correctness of the storage engine itself. Memtables earlier in order represent more recent mutations of the client data. If we didn't maintain an order, we'd be returning incorrect values of data when asked by a client. + +- Is the memory layout of the memtable efficient / does it have good data locality? (Think of how Byte is implemented and stored in the skiplist...) What are the possible optimizations to make the memtable more efficient? + - Skip lists' nodes are scattered so the memory layout is not cache friendly. We could do key prefix compression to save memory. + +- So we are using parking_lot locks in this tutorial. Is its read-write lock a fair lock? What might happen to the readers trying to acquire the lock if there is one writer waiting for existing readers to stop? + - Not fair. Writer can starve; we need to introduce explicit fairness if needed. + +- After freezing the memtable, is it possible that some threads still hold the old LSM state and wrote into these immutable memtables? How does your solution prevent it from happening? + - `state` lock will prevent this because we are taking a write lock while freezing which does the atomic swap of memtables. + +- There are several places that you might first acquire a read lock on state, then drop it and acquire a write lock (these two operations might be in different functions but they happened sequentially due to one function calls the other). How does it differ from directly upgrading the read lock to a write lock? Is it necessary to upgrade instead of acquiring and dropping and what is the cost of doing the upgrade? + - Acquiring and releasing increases the responsiveness of the engine as other threads can continue making progress. \ No newline at end of file diff --git a/mini-lsm/src/test_your_understanding/Week1/Day2.md b/mini-lsm/src/test_your_understanding/Week1/Day2.md new file mode 100644 index 00000000..a73462ee --- /dev/null +++ b/mini-lsm/src/test_your_understanding/Week1/Day2.md @@ -0,0 +1,35 @@ +### Test your understanding + +#### LSM-specific + +- What is the time/space complexity of using your merge iterator? + - Merge iterator is an iterator over the Memtable iterators. Memtable iterator is an iterator over SkipMap. Calling next() on SkipMap and + hence the Memtable is amortized to O(1). Calling next() on Merged iterators is O(log(M)), as it maintains a binary heap of all memtables (M). So merged iterator time complexity will be O(N*log(M)), where N is the total number of entries across all memtables, in the worst case. The space complexity of the merged iterator will be equivalent to the number of memtables, because the iterator itself just stores the reference + to individual memtable iterators. + +- If a key is removed (there is a delete tombstone), do you need to return it to the user? Where did you handle this logic? + - No, we don't have to return it. We are returning a None for non-existent keys from lsm_storage where it's handled. Memtable + stores an empty value for deleted entries so just returns that and isn't handled there. + +- If a key has multiple versions, will the user see all of them? Where did you handle this logic? + - If a key has multiple versions, our merged iterator's next() method handles it as it skips over the key for older memtables. + +- What happens if your key comparator cannot give the binary heap implementation a stable order? + - We may return old and incorrect values for a key. + +- Why do we need to ensure the merge iterator returns data in the iterator construction order? + - The iterator construction order specifies the "age" of memtables, where the earliest element is the latest. + +#### Rust-specific + +- Why do we need a self-referential structure for memtable iterator? + - Because we want the iterator to outlive the underlying Memtable or SkipMap. Without a self-referential structure, the borrow checker + will complain about lifetime errors, as it can't determine whether the iterator lives long enough. + +- Is it possible to implement a Rust-style iterator (i.e., next(&self) -> (Key, Value)) for LSM iterators? What are the pros/cons? + The scan interface is like fn scan(&self, lower: Bound<&[u8]>, upper: Bound<&[u8]>). How to make this API compatible with Rust-style range (i.e., key_a..key_b)? If you implement this, try to pass a full range .. to the interface and see what will happen. + The starter code provides the merge iter + +- If we want to get rid of self-referential structure and have a lifetime on the memtable iterator (i.e., MemtableIterator<'a>, where 'a = memtable or LsmStorageInner lifetime), is it still possible to implement the scan functionality? + What happens if (1) we create an iterator on the skiplist memtable (2) someone inserts new keys into the memtable (3) will the iterator see the new key? + - \ No newline at end of file diff --git a/mini-lsm/src/test_your_understanding/Week1/Day3.md b/mini-lsm/src/test_your_understanding/Week1/Day3.md new file mode 100644 index 00000000..3c1cba02 --- /dev/null +++ b/mini-lsm/src/test_your_understanding/Week1/Day3.md @@ -0,0 +1,39 @@ +### Test your understanding + +#### LSM-specific + +- What is the time complexity of seeking a key in the block? + - It's O(log(n)) where n is the number of keys/entries in the block. + +- Where does the cursor stop when you seek a non-existent key in your implementation? + - It stops at the key greater than the non-existent key. If the seeked key is greatest, + it will stop at the last element. + +- What is the endian of the numbers written into the blocks in your implementation? + - Bytes `put_u16` stores data in Big endian format. + +- Is your implementation prune to a maliciously-built block? Will there be invalid memory access, or OOMs, if a user deliberately construct an invalid block? + - We check for buffer overflows and invalid offsets to counter these issues. + - We check for block sizes before adding to prevent OOMs. + +- Can a block contain duplicated keys? + - Well, a block can. But if the block is always constructed from Memtable and merge iterators, then the merge iterator + will handle skipping the duplicate keys. + +- What happens if the user adds a key larger than the target block size? + - We don't accept the write and send a `false` back to the user. + +- Consider the case that the LSM engine is built on object store services (S3). How would you optimize/change the block format and parameters to make it suitable for such services? + - The target block size will be larger for object store services as most objects are large. We can avoid syscall and network overhead with a + larger block size. + - Since object stores are likely to span multiple blocks for each object, maybe the offsets/data split has more overhead. However, we still need + to handle small objects, so for now I'd keep the format as it is. + - We may want to also implement some compression to reduce transfer sizes. + +- Do you love bubble tea? Why or why not? + - Haha, I love the ones that are a little less sweet :-) + +#### Rust-specific + +- So Block is simply a vector of raw data and a vector of offsets. Can we change them to Byte and Arc<[u16]>, and change all the iterator interfaces to return Byte instead of &[u8]? (Assume that we use Byte::slice to return a slice of the block without copying.) What are the pros/cons? + - \ No newline at end of file