Can't find a way to trigger RocksDB cleanup more often

[AntonyLittle]

Description

I have noticed that, when RocksDB is initialised, it does a cleanup/compaction on existing data. I cannot, however, figure out how to make this happen while to service is running. Any ideas?

This document (https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide) mentions options.tll and options.periodic_compaction_seconds, both of which would be of use to us, but I cannot see how to access these settings through Streamiz.

[LGouellec]

Hey @AntonyLittle ,

TTL is not available with RocksDb through Streamiz for multiple reasons. The main one is RocksDb is not the source of through in Kafka Streams, the internal changelog topic does. So you can't remove deprecated data in RocksDb if the internal topic still contains the data.

Regarding options.periodic_compaction_seconds, this setting is not accessible easily , but you can override the parameter with a such custom rocksdb handler :

RocksDbConfigHandler = (store, options) =>
                {
                    var field = options.GetType().GetField("dbOptions", BindingFlags.NonPublic | BindingFlags.Instance);
                    Native.Instance.rocksdb_options_set_periodic_compaction_seconds(
                        (field.GetValue(options) as DbOptions).Handle, 10000);
                }