v0.3.0

This version release includes a number of changes that will greatly reduce G3 File sizes, and improve performance. This is mostly thanks to the G3SuperTimestream G3Object that Matthew wrote, which:

1. Fixes the 64-bit storage issue
2. Implements compression of timestreams

Performance will vary depending on a few factors, such as frame-size, number of detectors, how many are reading real data vs streaming constant values, but we predict that it will reduce file sizes by a factor of ~6 or so.

In order for this to be used, you need to also upgrade to sodetlib v0.3.0 (or greater) so that you can load data written in the new format.

Dockerhub page here

Updates

G3SuperTimestream

The biggest update in this version is the switch of the base data format from the G3TimestreamMap to G3SuperTimestream, which has the benefits of:

1. Being able to store 32 bit data, halving the size of G3 files
2. Being able to compress data with FLAC and BZIP, allowing us to reduce file-sizes by another factor of ~3 or so
3. General improved performance, see Matthew's PR or the so3g docs for more details

New rogue variables were added to control compression behavior:

• EnableCompression: Enables compression on serialization
• FlacLevel: Sets the flac level used
• Bz2WorkFactor: Sets the Bz2 work factor used by bzip
• DataEncodeAlgo: Sets which compression algorithms are used on detector data
• PrimaryEncodeAlgo: Sets which compression algorithms are used on Primary data
• TesBiasEncodeAlgo: Sets which compression algorithms are used on TES Bias data
• TimeEncodeAlgo: Sets which compression algorithms are used on timestamps
• BuilderEncode: Sets whether encoding occurs in the builder (as opposed to when frames are written to disk)
  Most of these people shouldn't really need to touch.

Default Meta-vars

A up-to-date meta_registers.yaml is now built into the docker and loaded by default. If a meta_registers file is specified in the sys-config, it will now update the values of the default. This way users don't also need to make sure their meta registers file is up-to-date along with the streamer version.

Publishing metadata of completed files

When a G3File is completed, a message is now published via the Pysmurf Publisher which can be processed by the pysmurf-monitor, allowing G3-Files to be handled by the SupRsync agent

open_g3stream register

An additional Rogue register called open_g3stream has been added that will tell the SessionManager to begin a new streaming session. This will allow us to stream data without relying on pysmurf's FileWriter.IsOpen register, which is only set when .dat files are being written, and allows us to stream g3 data without streaming to .dat.

Performance and Structural Improvements

Improved the structure of the Smurf-Builder's FlushStash function so that we can be sure it is only being called by a single thread. Also added compiler optimization which greatly improves Builder performance.

v0.2.0

Creating a new release because there have been a lot of changes recently:

Upgrades to pysmurf v5.0.2

This release contains the official upgrade to pysmurf v5.0.2. Check out the pysmurf release page if you're interested in details, but here's a summary of important fixes that this update should contain:

• Fix for dropped downsampled data frames
• Fix for TES Bias streaming for > 8 bias lines.
• Everything that comes with pysmurf v4.2 (find-freq speedup, etc.)

Local G3 streaming

The streamer will now write all streamed data locally to g3 files. Data will still be sent over the G3NetworkSender for use by lyrebird / other live-monitors, but we no longer need the smurf-recorder ocs agents to write the data to disk.

To set up:

1. Create a directory where you want g3 data to be stored. I've been using /data/so/timestreams on the smurf-server
2. Add that directory to the sys-config file with the g3_dir key. See here for example
3. There are a few extra metadata registers you need to add to the meta_registers.yaml file. You can just copy the file from the ucsd-k2so site-config directory into your own ocs-config-dir.

v0.1.0

Changes since the last release:

• Bump to pysmurf-server version v3.0.0, which is built on the new pysmurf version 4.2.0. This should add a lot of features and bug-fixes, including the unwrapper bug that has been crashing people's servers.
• Solved a bottleneck in the FlushStash that may have been throttling the data rate
• Added a primary data field to the G3Frames, smurf-header information
• various other bug fixes

Dockerhub page: https://hub.docker.com/layers/simonsobs/smurf-streamer/v0.1.0/images/sha256-7614b150e5a3e6a676a5d41377077e2cef69aa28dbc20167f945e4c3fa75922f?context=explore