Workload-aware Streaming State Management Testbed

This repository contains the testbed used in the paper 'In support of workload-aware streaming state management' (HotStorage 2020).

The testbed is based on Timely Dataflow and currently supports two key-value stores for backing up operator state: FASTER (via faster-rs) and RocksDB (via rust-rocksdb).

The testbed supports three basic state primitives:

1. ManagedCount (for counters)
2. ManagedValue (for arbitrary values)
3. ManagedMap (for unordered maps)

Pre-requisites for running experiments

All of our experiments require an installation of Rust 1.38.0 and the following dependencies

• CMake
• g++-7
• libaio-dev
• uuid-dev
• libtbb-dev

These dependencies are required to be able to compile FASTER.

Configuring state backends

The configuration files for FASTER and RocksDB instances are located inside the monolithic folder.

You can configure FASTER (hash index size and log size) using the faster.config file.

You can configure RocksDB (block size, LRU size, write buffer size, hash block index size) using the following files:

• rocksdb.config (for RocksDB with PUT/GET)
• rocksdbmerge.config (for RocksDB with MERGE in RANK queries)
• rocksdbmerge2.config (for RocksDB with MERGE in COUNT queries)

Running Nexmark queries

Each query can be run for a specified duration (in seconds) and with a given event generation rate (events/s).

For example, to run Nexmark Q3 for 1000 seconds with an input rate of 1M events/s using FASTER as the state backend:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries q3_faster

To run the same query using RocksDB as the state backend:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries q3_rocksdb

Running workload-aware Nexmark queries

Within the directory workload_aware run the following command:

$ cargo run --release -- --duration <duration> --rate <rate> --queries <query> 

where query is one of q3, q4, q5, q6, q7, q8

Running window queries

Window queries require two more arguments:

• window-slide: the size of the window slide in seconds
• window-slice-count: the number of slides in a window, i.e. the window size in seconds is window-slide*window-slice-count

For example, to run a COUNT aggregation on a 30s window with 1s slide using RocksDB (with PUT/GET) as the state backend:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries window_2a_rocksdb_count --window-slide 1 --window-slice-count 30

To run the same query using RocksDB (with MERGE) as the state backend:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries window_2b_rocksdb_count --window-slide 1 --window-slice-count 30

To run the same query using FASTER as the state backend:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries window_2_faster_count --window-slide 1 --window-slice-count 30

Running with multiple workers/processes

Timely Dataflow accepts configuration via arguments supplied at runtime. These can be passed by adding an extra -- between the line above and Timely's arguments.

For example, to run with four workers:

$ cargo run --release -- --duration 1000 --rate 1000000 --queries q3_faster -- -w 4

Explaining the output

This will produce output similar to the following:

    Finished release [optimized + debuginfo] target(s) in 0.03s
     Running `target/release/nexmark --rate 1000000 --duration 30 --queries q3_faster -- -w4`
latency_ccdf	122880	1	21
latency_ccdf	126976	0.9999997980769251	74
latency_ccdf	131072	0.9999990865384704	219
latency_ccdf	139264	0.9999969807692598	35299
latency_ccdf	147456	0.999657567310985	296241
latency_ccdf	155648	0.996809096184528	636340
latency_ccdf	163840	0.9906904423972073	793091
latency_ccdf	172032	0.9830645674705331	830141
latency_ccdf	180224	0.9750824425472842	838164
latency_ccdf	188416	0.9670231733940079	842592
latency_ccdf	196608	0.9589213273180641	846156
latency_ccdf	204800	0.9507852120116806	848651
latency_ccdf	212992	0.9426251063209125	850364
latency_ccdf	221184	0.9344485294764564	850920
latency_ccdf	229376	0.9262666064782057	851531
latency_ccdf	237568	0.9180788084800114	851461
latency_ccdf	245760	0.9098916835587338	851930
latency_ccdf	253952	0.9017000490221149	851918
latency_ccdf	262144	0.8935085298701103	851825
latency_ccdf	278528	0.8853179049488663	1703577
latency_ccdf	294912	0.8689373570294485	1703895
latency_ccdf	311296	0.8525537514177524	1703857
latency_ccdf	327680	0.8361705111906682	1703741
latency_ccdf	344064	0.8197883863481886	1703852
latency_ccdf	360448	0.803405194198027	1703642
latency_ccdf	376832	0.7870240212786151	1703881
latency_ccdf	393216	0.7706405502823024	1703869
latency_ccdf	409600	0.7542571946706039	1703739
latency_ccdf	425984	0.7378750890588934	1704149
latency_ccdf	442368	0.7214890411395285	1703737
latency_ccdf	458752	0.705106954758587	1703815
latency_ccdf	475136	0.6887241183776527	1703952
latency_ccdf	491520	0.6723399646890388	1703639
latency_ccdf	507904	0.6559588206157806	1704081
latency_ccdf	524288	0.6395734265425632	1704028
latency_ccdf	557056	0.6231885420847255	3407857
latency_ccdf	589824	0.5904206866305703	3407803
latency_ccdf	622592	0.5576533504071793	3407744
latency_ccdf	655360	0.5248865814914752	3408032
latency_ccdf	688128	0.4921170433450284	3407816
latency_ccdf	720896	0.4593495821216386	3407890
latency_ccdf	753664	0.42658140935979416	3407847
latency_ccdf	786432	0.39381365005948415	3407831
latency_ccdf	819200	0.3610460446053265	3407956
latency_ccdf	851968	0.3282772372281035	3407980
latency_ccdf	884736	0.29550819908165193	3407769
latency_ccdf	917504	0.2627411897813347	3407624
latency_ccdf	950272	0.22997557471177332	3408205
latency_ccdf	983040	0.1972043731038041	3407834
latency_ccdf	1015808	0.16443673880349288	3407742
latency_ccdf	1048576	0.1316699891185578	3407766
latency_ccdf	1114112	0.09890300866439415	6815950
latency_ccdf	1179648	0.033365028525336266	3445192
latency_ccdf	1245184	0.00023818269001747413	22930
latency_ccdf	1310720	0.00001770192290671228	758
latency_ccdf	1376256	0.000010413461438332101	524
latency_ccdf	1441792	0.000005374999948317308	314
latency_ccdf	1507328	0.000002355769208117604	242
latency_ccdf	1572864	0.000000028846153568786986	2
latency_ccdf	1638400	0.000000009615384522928995	0
summary_timeline	0	442368	720896	983040	1245184	1245184	1310720	1376256
summary_timeline	250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	750000000	425984	688128	950272	1179648	1245184	1310720	1310720
summary_timeline	1000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	1250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	1500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	1750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	2000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	2250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	2500000000	442368	720896	950272	1179648	1245184	1310720	1507328
summary_timeline	2750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	3000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	3250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	3500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	3750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	4000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	4250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	4500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	4750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	5000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	5250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	5500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	5750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	6000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	6250000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	6500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	6750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	7000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	7250000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	7500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	7750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	8000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	8250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	8500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	8750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	9000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	9250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	9500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	9750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	10000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	10250000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	10500000000	425984	688128	950272	1179648	1245184	1376256	1638400
summary_timeline	10750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	11000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	11250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	11500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	11750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	12000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	12250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	12500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	12750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	13000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	13250000000	425984	688128	950272	1179648	1245184	1245184	1376256
summary_timeline	13500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	13750000000	425984	688128	950272	1179648	1245184	1310720	1376256
summary_timeline	14000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	14250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	14500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	14750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	15000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	15250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	15500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	15750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	16000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	16250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	16500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	16750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	17000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	17250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	17500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	17750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	18000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	18250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	18500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	18750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	19000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	19250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	19500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	19750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	20000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	20250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	20500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	20750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	21000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	21250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	21500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	21750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	22000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	22250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	22500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	22750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	23000000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	23250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	23500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	23750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	24000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	24250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	24500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	24750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	25000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	25250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	25500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	25750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	26000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	26250000000	425984	688128	950272	1179648	1245184	1245184	1245184
summary_timeline	26500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	26750000000	425984	720896	950272	1179648	1245184	1310720	1376256
summary_timeline	27000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	27250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	27500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	27750000000	425984	688128	950272	1179648	1245184	1245184	1376256
summary_timeline	28000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	28250000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	28500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	28750000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	29000000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	29250000000	425984	688128	950272	1179648	1245184	1376256	1638400
summary_timeline	29500000000	425984	688128	950272	1179648	1245184	1245184	1310720
summary_timeline	29750000000	425984	688128	950272	1179648	1245184	6291456	6553600

The output is a list of tab-separated values on stdout.

• latency_ccdf 229376 0.9262666064782057 851531: latency CCDF value, 851531 measurements, smaller than 0.9262666064782057% of all measurements, latency 229376ns.
• summary_timeline 1250000000 425984 688128 950272 1179648 1245184 1245184 1310720: Some percentiles at time 1250000000ns: 25%, 50%, 75%, 99%, 99.9%, max in nanoseconds.