Manual Compaction

Compaction can be triggered manually by calling the DB::CompactRange or DB::CompactFiles method. This is meant to be used by advanced users to implement custom compaction strategies, including but not limited to the following use cases:

1. Optimize for read heavy workloads by compacting to lowest level after ingesting a large amount of data
2. Force the data to go through the compaction filter in order to consolidate it
3. Migrate to a new compaction configuration. For example, if changing number of levels, CompactRange can be called to compact to bottommost level and then move the files to a target level

The example code below shows how to use the APIs.

Options dbOptions;

DB* db;
Status s = DB::Open(dbOptions, "/tmp/rocksdb",  &db);

// Write some data
...
Slice begin("key1");
Slice end("key100");
CompactRangeOptions options;

s = db->CompactRange(options, &begin, &end);

Or

CompactionOptions options;
std::vector<std::string> input_file_names;
int output_level;
...
Status s = db->CompactFiles(options, input_file_names, output_level);

CompactRange

The begin and end arguments define the key range to be compacted. The behavior varies depending on the compaction style being used by the db. In case of universal and FIFO compaction styles, the begin and end arguments are ignored and all files are compacted. Also, files in each level are compacted and left in the same level. For leveled compaction style, all files containing keys in the given range are compacted to the last level containing files. If either begin or end are NULL, it is taken to mean the key before all keys in the db or the key after all keys respectively.

If more than one thread calls manual compaction, only one will actually schedule it while the other threads will simply wait for the scheduled manual compaction to complete. If CompactRangeOptions::exclusive_manual_compaction is set to true, the call will disable scheduling of automatic compaction jobs and wait for existing automatic compaction jobs to finish.

DB::CompactRange waits while compaction is performed on the background threads and thus is a blocking call.

The CompactRangeOptions supports the following options -

• CompactRangeOptions::exclusive_manual_compaction When set to true, no other compaction will run when this manual compaction is running. Default value is true
• CompactRangeOptions::change_level,CompactRangeOptions::target_level Together, these options control the level where the compacted files will be placed. If target_level is -1, the compacted files will be moved to the minimum level whose computed max_bytes is still large enough to hold the files. Intermediate levels must be empty. For example, if the files were initially compacted to L5 and L2 is the minimum level large enough to hold the files, they will be placed in L2 if L3 and L4 are empty or in L4 if L3 is non-empty. If target_level is positive, the compacted files will be placed in that level provided intermediate levels are empty. If any any of the intermediate levels are not empty, the compacted files will be left where they are.
• CompactRangeOptions::target_path_id Compaction outputs will be placed in options.db_paths[target_path_id] directory.
• CompactRangeOptions::bottommost_level_compaction When set to BottommostLevelCompaction::kSkip, or when set to BottommostLevelCompaction::kIfHaveCompactionFilter (default) and a compaction filter is **not** defined for the column family, the bottommost level files are not compacted. BottommostLevelCompaction::kForce can force the compaction, which could also avoid trivial move to the bottommost level.
• CompactRangeOptions::allow_write_stall When set to true, it will execute immediately even if doing so would cause the DB to enter write stall mode. Otherwise, it'll sleep until load is low enough. Default value is false
• CompactRangeOptions::max_subcompactions If > 0, it will replace the option in the DBOptions for this compaction.
• CompactRangeOptions::full_history_ts_low Set user-defined timestamp low bound, the data with older timestamp than low bound maybe GCed by compaction.
• CompactRangeOptions::canceled Allows cancellation of an in-progress manual compaction. Cancellation can be delayed waiting on automatic compactions when used together with exclusive_manual_compaction == true.
• CompactRangeOptions::blob_garbage_collection_policy If set to kForce, RocksDB will override enable_blob_file_garbage_collection to true; if set to kDisable, RocksDB will override it to false, and kUseDefault leaves the setting in effect. This enables customers to both force-enable and force-disable GC when calling CompactRange. Default value is kUseDefault.
• CompactRangeOptions::blob_garbage_collection_age_cutoff If set to < 0 or > 1, RocksDB leaves blob_file_garbage_collection_age_cutoff from ColumnFamilyOptions in effect. Otherwise, it will override the user-provided setting. This enables customers to selectively override the age cutoff. Default value is -1 (do not override).

CompactFiles

This API compacts all the input files into a set of output files in the output_level. The number of output files is determined by the size of the data and the setting of CompactionOptions::output_file_size_limit. This API is not supported in ROCKSDB_LITE.