Throttle late packet messages #4
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
... in the block. The disk thread uses the same bcnt to map the BDA baseline-times to the hdf5 file.
Otherwise it gets too noisy
And don't allow start a recording by simply increasing the NFILES without a trigger
… valid Because each packet contains 3 ants and they may not all be real
…than are valid" This reverts commit 0316c7d.
Useful for sanity
Not clear how these can be different in a working system at present
2 workers, and one parent
For testing, don't bother to throttle the output packets.
NB: hardcoded N_ANTS still exists in the template hdf5 header generation script
... to us the 16 sec of integration time for sending data, instead of the 2sec it takes to fill the buffer, separating the BDA averaging and data output to the catcher machine.
... catcher writes 16 sec integration files. Changed packet reception and data writing to accomodate BDA parameters.
... scripts to check output of X-engs and generate fake input to the catcher.
... thread to terminate a hashpipeline for testing. Takes the last databuf available and sets it free for the pipeline to continue.
Make repo python3-only
Network issues
Minor variable declaration that didn't get added to pull request #42
Update hera_catcher_net_thread_bda.c
Changed affinity of NET CPU in px* machines in paper_init.sh
Fix nsamples array in data files
…ta catcher with all snaps working.
Tweaks made by Aaron Parsons to absorb a higher data rate into the da…
Previously, each late packet was logged because late packets was assumed to be a rare occurrence. Unfortunately, due to not-yet-understood circumstances, the HERA X engines sometimes receive "late" packets from some of the SNAPs, sometimes many many late packets from some of the SNAPs. Logging all of these late packets causes the log files to grow until eventually the file system is filled up, which further impacts observing. These messages are now logged in a still generous, but much more constrained quantity. Up to 5,000 of these "late packet" messages will be logged within an hour from the first such log message. After that hour has elapsed, the next such log message will start a new hour long window. This will result in a max of 120,000 such log messages in a 24 hour period (per X engine), which is far fewer than the 3+ million such log messages per X engine that led to a full file system. The threshold of 5,000 was chosen because there seemed to be some periodicity in the neighborhood of 1024 occurrences. The limit of 5,000 will allow for the capture of several such cycles per "burst".
The reduced threshold of 120 messages per burst should still be plenty for diagnostic purposes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is intended to prevent the disk from filling up when a "late packet storm" occurs. See the commit log message for more details.