Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Packaging Revamp #1024

Draft
wants to merge 24 commits into
base: master
Choose a base branch
from
Draft

Data Packaging Revamp #1024

wants to merge 24 commits into from

Conversation

kmharrington
Copy link
Member

@kmharrington kmharrington commented Nov 6, 2024 •
edited
Loading

Still a draft because work isn't completed. But this is currently running on the site machines.

New features:

  • datapkg_completion module has a DataPackage class I started to working with both G3tSmurf, G3tHK, and Imprinter at the same time to go through and do the checks necessary to make sure we're ready to delete data from each timecode.
  • updates to Imprinter to check with librarian about the status of off-site books
  • Closes Incomplete observation finding doesn't work for LAT #938
  • Significantly reduces the size of the g3thk databases, now we have file entries by only the necessary smurf-related agents get added to hkagents and hkfields
  • Add ability to bind books without HK files is flags are set correctly

sanahabhimani and others added 24 commits May 28, 2024 04:11
@sanahabhimani
G3tHK Update: Trim
00bb68d 
Updates g3thk to only query fields for agents from the site-pipeline configs;
i.e., pysmurf monitor and supersync to aid the data packaging tools from
the level2 side, and there fore reduce g3thk db size
@sanahabhimani
actually push non hard-coded version g3thk trim
de6783a
@sanahabhimani
add docstring and remove prints and commented lines
1e9ecee
@sanahabhimani
push non hardcoded version for getting all iids and not the first one…
f13e288 
… in the config file
@kmharrington
sqlite int fix and use in_ for queries
b332164
@kmharrington
add all_slots as a property. closes #938
8f91660
@kmharrington
add method to check if librarian has the book offsite
b8b11b7
@kmharrington
add lvl2 data folder to imprinter and generate smurf book file lists
8461a4f
@kmharrington
add many functions for checking completion of a time period
c053174
@kmharrington
Squashed commit of the following:
56c91a1 
Add DataPackaging class for going throught, checking completion and running
time-code based deletion. HK and verification through the librarian are
optional.
@kmharrington
Merge remote-tracking branch 'origin/g3thk-trim' into dpkg-revamp
5913c5d
@kmharrington
updates to allow for book binding without HK data
cb95b77
@kmharrington
update g3thk connection
29f7bca
@kmharrington
set new hk connection to be general
8a9fefd
@kmharrington
account for hk databases without detector data
c38faaf
@kmharrington
add debug message
7f6a7b5
@kmharrington
fix debug message
8404741
@kmharrington
fix DataPackaging module to deal with imprinters with no detector data
eb459d4
@kmharrington
updated to start trying to scriptify the cleanup
a975b7f
@kmharrington
change logger name
e727fb1
@kmharrington
finish variable name changes
96f8d7c
@kmharrington
fix typos and update logging levels
678692b
@kmharrington
more log updates
cc2d033
@kmharrington
move per-book checks into imprinter
73bc9b8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Incomplete observation finding doesn't work for LAT
2 participants
@kmharrington @sanahabhimani