[GSoC 2023]Student Blog for FASER Real Time Compression Project - SumalyoDatta #1458
Conversation
| FASER has a trigger-based data acquisition model, and the DAQ software is developed using the DAQling framework. | ||
| >> DAQling is an open-source lightweight C++ software framework that can be used as the core of data acquisition systems of small and medium-sized experiments. It provides a modular system in which custom applications can be plugged in. It also provides the communication layer based on the widespread ZeroMQ messaging library, error logging using ERS, configuration management based on the JSON format, control of distributed applications and extendable operational monitoring with web-based visualization. | ||
|
|
||
| You can read more about the strategy in the [paper published by the FASER collaboration](https://arxiv.org/abs/2110.15186). |
There was a problem hiding this comment.
I would replace "strategy" with "FASER TDAQ design".
| The [FASER experiment](https://faser.web.cern.ch/index.php/) is an LHC experiment located in a tunnel parallel to the LHC ring. It is considerably smaller and low-budget compared to the titan LHC experiments such as ATLAS and CMS. The experiment seeks to detect new long-lived particles that have travelled half a kilometre from the LHC proton-proton collision site at the centre of the ATLAS experiment, escaping the ATLAS detector undetected. During proton collisions at the LHC, the FASER experiment records up to 1500 events per second using an open-source data acquisition (DAQ) software framework developed at CERN. The DAQ software receives dedicated data fragments from subcomponents on the detector, packs them into a single event and writes completed events to a file. The assigned total storage space for the experiment was, however, quickly met and already resulted in a doubling of the requested storage space. The challenge was to develop a compression engine that would compress data in real-time, __introducing minimal latency and performance overheads__. The compression engine would also have to decompress events transparently for data reconstruction for physics analysis. <p> | ||
|
|
||
| FASER has a trigger-based data acquisition model, and the DAQ software is developed using the DAQling framework. | ||
| >> DAQling is an open-source lightweight C++ software framework that can be used as the core of data acquisition systems of small and medium-sized experiments. It provides a modular system in which custom applications can be plugged in. It also provides the communication layer based on the widespread ZeroMQ messaging library, error logging using ERS, configuration management based on the JSON format, control of distributed applications and extendable operational monitoring with web-based visualization. |
There was a problem hiding this comment.
If this is quoted from somewhere, maybe add a link to the document as reference?
| The approach for finding the best compressor is described below | ||
| - Events were divided into a set of 10 classes based on event size (Class 0 having events of smallest size) | ||
| - For each event class, the average compression speed and compression ratio were calculated (for each compressor) | ||
| - The resulting points were plotted on a graph, and this helped to visualize the tradeoff. The compressor configuration offering the highest average compression ratio at the highest compression speed was considered optimal. |
There was a problem hiding this comment.
A bit nitpicky since I know what you mean: Since there is a trade-off between compression ratio and speed, we will not get an algorithm that will achieve both highest compression and highest speed. I would rephrase by referring to your "acceptance region" in fig 2 and something like "The compressor configuration configuration offering the highest compression speed at an acceptable compression ratio" (or vice versa).
very nice, thanks :) |
Updating broken link
|
@cantel Seems like your comments were not followed up on, should we merge anyway or close? |
| @@ -67,7 +73,7 @@ The logs and the python-based analysis notebooks can be found at [this repo](htt | |||
| The approach for finding the best compressor is described below | |||
| - Events were divided into a set of 10 classes based on event size (Class 0 having events of smallest size) | |||
| - For each event class, the average compression speed and compression ratio were calculated (for each compressor) | |||
| - The resulting points were plotted on a graph, and this helped to visualize the tradeoff. The compressor configuration offering the highest average compression ratio at the highest compression speed was considered optimal. | |||
| - The resulting points were plotted on a graph, and this helped to visualize the tradeoff. A compression ration of about 2 (or 50% compression ) and a speed more that 40 MB/s was considered as acceptable performance (as denoted in the diagram above).The compressor configuration offering the highest average compression ratio at the highest compression speed (in the acceptable performance region of the graph) was considered optimal. | |||
There was a problem hiding this comment.
Many thanks, @Sumalyo . Since it's such a nice blog entry, I'm adding one last comment on your last commit as it includes some grammar mistakes: "compression ration" ->"compression ratio", "and a speed more that 40 MB/s" -> "and a compression speed of more than 40 MB/s"
There was a problem hiding this comment.
Hi Claire,
I have made the changes that you have suggested
|
No problem, thanks for the quick reaction and the nice blog post! |
My student blog for GSoC 2023
Project - Real-time lossless data compression for the FASER experiment
Mentors: Claire Antel and Brian Petersen
@cantel, Could you please review?