The following data sets contain the raw performance data and metadata of the benchANT blog post Apache Cassandra version monitoring benchmark.
Each folder contains the data of a single benchmark run, i.e. one data point. Each benchmark configuration is executed three times, i.e. each benchmark configurations is based on three data points.
In order to ensure full transparency and reproducibility, each data folder contains benchmark configuration data, performance data, monitoring data, cloud provider metadata, VM metadata and DBMS configuration data.
In addition the aggregation.xlsx provides an abstracted view over all data points.
All configurable benchmark parameters are defined in the evaluationScenario.json.
The benchANT_versions contains the used versions of the benchANT software components to execute the benchmarks.
The execution logs of the individual benchmark steps are contained in airflowTaskInstanceDetails.json.
The raw performance data output of the YCSB is contained in the 0_load.txt for the LOAD phase and in the 0_run.txtfor the RUN phase. The database ranking data only considers the RUN phase.
In addition, the runtimeDataframe.xlsx represents a cleaned time-series of the LOAD and RUN phase performance data.
The validate.json provides a validation overview of the raw benchmark results.
The DBMS cluster and the benchmark instances are monitored with Telegraf and the data is stored in InfluxDB.
A full snapshot of the monitoring data of each run is contained in the influx_data.zip file.
The time frame of the RUN phase for the relevant metrics is extracted in the dbmsMetrics.xlsx.
The cloud provider metadata for the DBMS deployment is contained in the dbms_data_resources.json / dbms_management_resources.json and for the benchmark deployment in the benchmark_resources.json.
The VM metadata for the DBMS deployment is contained in the dbms_data_hardware_facts.json / dbms_management_hardware_facts.json and for the benchmark deployment in the benchmark_hardware_facts.json.
For each DBMS, relevant configuration files and cluster states are stored before executing the workload.
DBMS-specific files are contained in each folder, e.g. postgresql.conf for PostgreSQL DBMS deployments.
In case of questions or feedback on the data feel free to reach out to [email protected]
@inproceedings{10.1145/3491086.3492473,
author = {Seybold, Daniel and Domaschka, J\"{o}rg},
title = {Benchmarking-as-a-Service for Cloud-Hosted DBMS},
year = {2021},
isbn = {9781450391542},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3491086.3492473},
doi = {10.1145/3491086.3492473},
booktitle = {Proceedings of the 22nd International Middleware Conference: Demos and Posters},
pages = {12–13},
numpages = {2},
keywords = {cloud, DBMS, performance, scalability, benchmarking-as-a-service},
location = {Virtual Event, Canada},
series = {Middleware '21}
}
@phdthesis{seybold2021automation,
title={An automation-based approach for reproducible evaluations of distributed DBMS on elastic infrastructures},
author={Seybold, Daniel},
year={2021},
school={Universit{\"a}t Ulm}
}