Skip to content

v0.3.0

Latest
Latest
Compare
Choose a tag to compare
@jlashner jlashner released this 05 Nov 16:24
· 28 commits to master since this release
v0.3.0
6a254d0

This version release includes a number of changes that will greatly reduce G3 File sizes, and improve performance. This is mostly thanks to the G3SuperTimestream G3Object that Matthew wrote, which:

  1. Fixes the 64-bit storage issue
  2. Implements compression of timestreams

Performance will vary depending on a few factors, such as frame-size, number of detectors, how many are reading real data vs streaming constant values, but we predict that it will reduce file sizes by a factor of ~6 or so.

In order for this to be used, you need to also upgrade to sodetlib v0.3.0 (or greater) so that you can load data written in the new format.

Dockerhub page here

Updates

G3SuperTimestream

The biggest update in this version is the switch of the base data format from the G3TimestreamMap to G3SuperTimestream, which has the benefits of:

  1. Being able to store 32 bit data, halving the size of G3 files
  2. Being able to compress data with FLAC and BZIP, allowing us to reduce file-sizes by another factor of ~3 or so
  3. General improved performance, see Matthew's PR or the so3g docs for more details

New rogue variables were added to control compression behavior:

  • EnableCompression: Enables compression on serialization
  • FlacLevel: Sets the flac level used
  • Bz2WorkFactor: Sets the Bz2 work factor used by bzip
  • DataEncodeAlgo: Sets which compression algorithms are used on detector data
  • PrimaryEncodeAlgo: Sets which compression algorithms are used on Primary data
  • TesBiasEncodeAlgo: Sets which compression algorithms are used on TES Bias data
  • TimeEncodeAlgo: Sets which compression algorithms are used on timestamps
  • BuilderEncode: Sets whether encoding occurs in the builder (as opposed to when frames are written to disk)
    Most of these people shouldn't really need to touch.

Default Meta-vars

A up-to-date meta_registers.yaml is now built into the docker and loaded by default. If a meta_registers file is specified in the sys-config, it will now update the values of the default. This way users don't also need to make sure their meta registers file is up-to-date along with the streamer version.

Publishing metadata of completed files

When a G3File is completed, a message is now published via the Pysmurf Publisher which can be processed by the pysmurf-monitor, allowing G3-Files to be handled by the SupRsync agent

open_g3stream register

An additional Rogue register called open_g3stream has been added that will tell the SessionManager to begin a new streaming session. This will allow us to stream data without relying on pysmurf's FileWriter.IsOpen register, which is only set when .dat files are being written, and allows us to stream g3 data without streaming to .dat.

Performance and Structural Improvements

Improved the structure of the Smurf-Builder's FlushStash function so that we can be sure it is only being called by a single thread. Also added compiler optimization which greatly improves Builder performance.

Assets 2
Loading