QLF v0.7.2

Bug fix

QL update (#308)

QLF v0.7.1

Bug fixes

fixing monitor bug and updating desispec (#307)
update ql to current master (#306)
adding backend docker dependency (#305)

QLF v0.7.0

Features

QL update (#302)
Removing docker backend-frontend dependency (#304)
Test jupyter (#299)
adding docker image build to jenkins (#300)
Qa plots refactor (#298)
Survey report query optimization (#297)
Deployment kubernetes/nersc (#295)
Updating desispec (#284)
Check view files (#296)
Adding checkhdu (#294)
Changing header height (#291)
Include name in Help Credits (#289)

Bug fixes

Colorbar (#303)
Adding django debug and minor fixes (#301)
Fixing spectra and global fibers on fake generated exposures (#292)
Fixing plot size (#290)

Future Implementations

Responsive plots
Include SNR, TRACE SHIFTS and PSF FWHW options in Observing Conditions
Change meaning of OBJTYPE (TGT, SKY, BAD instead of ELG, LRG, QSO, ...)
Create tutorials
Time series QA optimization
DELTAMAG and CHECKARC there are still problems to be fixed with QL team

QLF v0.6.0

Features

Implementing the Tutorials screen (#288)
Tooltip implementation to explain date period (#286)
Responsive layout (#261) and monitor(#247)
Spectra viewer (#262)
Update desispec qlconfig (#253)
Data simulations (#273)
Removing regression, histograms and centering plots (#274)
Changing Step and QA names (#272)
Adding warning QA tick (#271)
Flat plots (#269)
Observing history regression (#254)
Organizing comments in info tables (#258)
Removing time series tabs (#259)
Using MJD on time series x axis (#260)
Adaptable plots (#249)
Survey Report initial load (#248)
Changing QA description (#242)
Time series changes (#246)
Rearranging plots layout (#244)
Added box_zoom in bokeh tools (#245)

Bug fixes

Tooltip placement settings Date caption (#287)
Fixing date picker and layout (#285)
Fix arc xwsigma (#283)
Fix Survey Report and spectrograph selection (#280)
Fix SNR obj (#281)
Correct aspect ratio and axes orientation (#282)
Fixing objsorter problems and update ref key (#277)
Fixing fail QA color (#278)
Fix merge issues (#276)
Fixing datetime format (#270)
Minor layout fixes (#263)
Fix amp plots (#257)
Validate submit button (#256)
Remove confusing plots (#252)

Future Implementations

Responsive plots
Change arm name from b to g
Include SNR, TRACE SHIFTS and PSF FWHW options in Observing Conditions
Change meaning of OBJTYPE (TGT, SKY, BAD instead of ELG, LRG, QSO, ...)
Create tutorials

QLF v0.5.0

Features

Submit Button Controls (#234)
Adding arm selection to global viewers (#240)
Layout by flavor (#233)
Mapping layout by flavor (#229)
Finished the integration from pipeline with main json file. (#236)
Get json chunk by camera and data range (#230)
NERSC deploy changes (#211)
Removing QA table and adding fibermap to database (#224)
Observing Conditions Implementation (#227)
Trend Analysis implementation (#218)
Survey Report Implementation (#219)
Resize Survey Report (#232)
Inclusion of footprint and statistics URLs in Survey Reports (#237)
Adding footprint object type count in Survey Report (#222)
Showing footprint on first load Survey Reports (#239)
Adding Observing Conditions and Trend Analysis plots (#223)

Bug fixes

QLF unused code cleanup (#243)
Fixing object statistics table (#241)
Typo fix (#238)
Disabling backend tests (temporary) (#235)
Fixing jenkins backend test (#231)
Fix monitor stages (#221)

QLF v0.4.0

Features

Change in Steps Tooltip (#212)
Adding desi footprint (#209)
Adding Regression and Time Series (backend) in Trend Analysis (#204)
Inclusion of the SNR in Processing History(View) (#203)
Mocking data (#202)
Change in date format (#196)
Inclusion of the Focus in Processing History(View) (#199)
Adding SNR backend (#201)
Removing configuration file (#191)
Inclusion of the Fibers in Processing History(View) (#189)
Renaming Sky -> Observing and CCDs -> View (#195)
Changing the alert sound and including the MUTE option (#193)
Adding LIneA logo in footer (#192)
Adding Processing History log viewer (#188)
Monitor QL critical errors (#187)
Showing observing history by night (#185)
Changing petals colors (#184)
Adding QLF offline mode (#183)
Store the output(product log) of each camera. (#179)
Adding status filter (#180)
Bokeh optimizations (#175)
Configures the parallel processes number in environments with low processing power (#167)
Send notification on QA alarm or warning (#174)

Bug fixes

Fixing bokeh plots layout (#217)
Amending key tables (#216)
Globalview changes (#215)
Metric alert tables (#214)
Fix realtime process (#213)
removing unused file and fixing fits to png (#210)
mergedqa dictionary read from get_merged_qa_scalar_metrics (#208)
reading qa merged status keys (#207)
Fix merged (#205)
Change LIneA logo, css firefox and centralize logo DESI vertically (#198)
Fixing desimodel data (#197)
Beginning code cleanup (#190)
Correction in icon position Info (#194)
Removed links creation to calibration files. (#186)
QL configuration filename update (#182)
Fix history dateobs ordering (#178)
Updating desispec to use qlstatus changes (#177)
Notification QA fix (#176)

Future implementations

• Observing Conditions
• Trend Analysis
• Survey Reports

QLF v0.3.1

Bug Fix

Notification fix #173

QLF v0.3.0

This release contains all changes made during the mock observing (#166) a well as all the changes listed bellow.

Features

• Monitor fixes (#172)
• Processing history changes (#171)
• Modify ingestion to add the ql-mergedQA. (#170)
• Changing TemplateNight and TemplateExpid values from the QL configura… (#158 )
• Showing all calibration files on configuration (#160)
• Adding aborted process info to pipeline (#152)
• Adding Ql configuration tabs (#153)
• QL pipeline integration (#147)
• Adding configuration edit (#149)
• Adding new clean disk layout (#148)
• Reset monitor screen, comments updating realtime, flavor fix and abor…(#146)
• ICS integration (#145)
• Adding processing history comments (#132)
• Adding petals selection to configuration (#136)
• New fits preview interface (#135)
• Adding history date range selection limits (#133)
• Frontend layout changes (#134)
• Adding disk space alarm notification (#137)
• Updating to desispec 0.21.0 (#126)
• Preparing docker to deploy using hub.docker (#131)
• Adding new development environment (#128)
• showing calibration on configuration screen (#129)

Bug Fixes

• Kpno updates (#166)
• Fix peakcount plot (#169)
• Fix plots (#168)
• Modified the directory structure in $DESI_SPECTRO_DATA. (#165)
• Kpno fixes (#164)
• Using current host user uid and group in docker container (#163)
• Fixed bug to get the last exposure used as template. (#161)
• Fix plots (#162)
• Fix plots (#157)
• Petals coloring realtime from log (#159)
• Fix plots (#155)
• Fixing png preview (#154)
• Plot tweak (#151)
• Investigating error at the end of the steps. (#144)
• Update qakeys (#141)
• Updating python and numpy version (#142)
• Removing the removal of the exposures at the beginning of the monitor… (#140)
• Fixing 0.21 update layout issues (#139)
• Removed unnecessary functions. (#138)

Future Implementations

• Send notifications on QA error
• Responsive plots
• Trend Analysis
• Product Logs
• User roles
• JSON files ingestion
• Manual mode
• Notification sound bug fix
• Websocket problem when running 30 cameras

QLF v0.2.0

Features

• Update frontend deployment (#125)
• Exposure generator improvements (#124)
• Adding notifications (#121)
• Adding google analytics (#120)
• Adding new configuration layout (#117)
• New exposure fields: program and airmass (#118)
• Coloring QAs realtime (#114)
• Adding version date and fixing date input layout (#116)
• Change monitor log (#115)
• New date picker (#113)
• Integration with new dataset and update desispec to 0.20.0 tag. (#104)
• Deleting processed exposures (#112)
• Deployment by git tag (#110)
• Update bokeh 0.12.16 (#109)
• Changing page titles and adding home button to appbar (#108)
• Hiding table columns (#102)
• Showing calibration on configuration screen (#129)
• Adding fits to png integration (#130)

Bug fixes

• Update qakeys (#127)
• Fixing layout for different screen size (#123)
• Fix color qa bug (#119)
• Date range and minor style fixes (#111)

Future Implementations

• Updating to desispec 0.21.0 #126
• Adding new development environment #128
• Notifications backend
• Ingestion using merged QAs
• Exposure Comments

QLF v0.1.0

As this is the first Release Notes we will detail all current front-end and back-end features.

Front-end

Monitor

Real-time app that controls and presents current system data

Features

• Start/Stop pipeline execution.
• Information about current exposure being processed.
• Follow evolution of pipeline steps for each camera.
  • Per camera log by clicking on the + on each camera process evolving line.
• Log highlights pipeline’s checkpoints and run-time.
• QA information for each PA displayed as tooltip when mouse over a petal.
• Monitor new exposures as they are discovered.

Processing History

Lists of all exposures processed by the pipeline.

Features

• List 10 most recent processes (Updated real time)
• History containing all exposures processed.
  • Can be filtered by exposure start and end date. (Not updated in real time)
  • Can be ordered by Process Id, Process Date, Exp Id, Tile ID, Obs date, Obs Time, RA, DEC, Exp Time
• QA column ✅ (or ❌) for each exposure links to QA screen app.

QA

Shows QA tests summary

Features

• Process ID, Exposure ID, MJD and Date for the selected exposure
• Each camera petal divided by PA results are represented by colors
  • Grey - when there is a QA not found
  • Red - alarm in a QA test
  • Yellow - warning in a QA test
  • Green - all QA tests in the step passed
• Drill down by clicking a petal (except when grey)
  • Metrics are shown by either red or green, respectively failure or success
  • Graphs are shown with a short description
  • All display for each step, spectrograph, and arm selection

Observing History

Lists of all exposures available (and respective last QA results when processed)

Features

• List 10 most recent exposures (updated in real time).
• History containing all exposures.
• They can be filtered by exposure start and end date. (not updated in real time).
• It can be ordered by Exp Id, Tile ID, Obs date, Obs TIme, RA, DEC, Exp Time.
• QA column ✅ (or ❌) for each exposure links to QA screen app (when process is available).
• Inside the history tab is possible to reprocess a selected exposure and add it to the end of the pipeline processing queue by selecting and clicking Submit.
• Filter flavour

Configuration

Features

• View QLF configuration
• View QL configuration

Backend

QLF Pipeline

Daemon running QL.

Features

• Runs QL using a configuration
• Ingest QA files to database
• Display QA scalar metrics

Django API

Administrative back-end managing Front-end and Pipeline connections

Features

• Provides an API
• Connects to QLF Daemon
• Manages Front-end websocket connections and REST services
• Connects to databases (currently using only a local postgres database)

Bokeh Server

Features

• Generate QA graphs for drill downs

Future Implementations

• Trend Analysis
• Afternoon Planning
• Sky Conditions
• Survey Reports
• Create and set a configuration
• PNG images for exposures
• History tables
• customize table columns
• Filter/order by Program, QA (status) and flavour
• Edit QL and QLF configuration
• ICS interface connection
• Implement manual mode (in contrast to automatic mode using daemon)
• Offline results display
• Implement multi-user management. Right now, only a user should use QLF at a time