DM-46414: Add zernikes astropy table output from CalcZernikesTask

[jmeyers314]

This combines the current outputs from outputZernikesRaw and outputZernikesAvg into a single new dataset type, using astropy.table.QTable as the storage class. Some features:

• There are individual named columns for each Zernike coefficient, so we can easily adapt this to sparse indexing in a future commit.
• The QTable tracks units.
• Each row is a different donut pair (we'll have to adapt in the future when we get tasks for running on a single side of focus).
• There's a bonus row with a label "average" to place what used to be in outputZernikesAvg.

Here's a quick preview of the format:

 label  used      intra_field [x, y]           extra_field [x, y]      ...         Z26                 Z27                 Z28        
                         deg                          deg              ...          nm                  nm                  nm        
------- ---- ---------------------------- ---------------------------- ... -------------------- ------------------ -------------------
average True                   (nan, nan)                   (nan, nan) ...   2.3800891163369617  41.22152418083769 -35.277507712739755
  pair1 True    (-0.05821145, 0.02770254)    (-0.05815588, 0.02764697) ...   2.9741328684160266  47.96728907450128 -51.641467608636205
  pair2 True (-0.03042558, -8.335776e-05) (-0.03042558, -8.335776e-05) ...    4.752704920327202 33.015250747032304 -40.203176051984826
  pair3 True     (0.02509067, 0.05548844)      (0.02509067, 0.0553773) ...   2.7752296616889507 33.957946113380295 -38.249810630913444
  pair4 True     (0.05287657, 0.02770254)       (0.052821, 0.02764697) ...  0.29520949398193513  40.05030889181167 -24.009861860875446
  pair5 True     (0.02509068, 0.02770256)     (0.02509068, 0.02764699) ...   3.3179267030988275   42.3718751313613 -27.333083707839172
  pair6 True  (0.05287658, -8.335772e-05)  (0.05282101, -8.335772e-05) ...   -2.446922617228806   43.8753425268592 -29.423867896649814
  pair7 True (0.02514626, -8.3357765e-05) (0.02509069, -8.3357765e-05) ...   0.5340847698338642  44.76449641394313 -37.178195535420855
  pair8 True    (0.05287657, -0.02786926)      (0.052821, -0.02786926) ...  -3.4206912611860387   46.0229076067933 -17.539900548414202
  pair9 True    (0.02514626, -0.02786927)    (0.02509068, -0.02786927) ...    5.105795693358347  48.71401822551286  -20.10808049221983
 pair10 True    (0.05287654, -0.05559955)    (0.05282096, -0.05559955) ...  -3.4056012162811595  45.10249008751407  -24.72798750910216
 pair11 True    (0.02514624, -0.05565516)    (0.02509067, -0.05559959) ...  0.29033472810139616  51.69241538940008 -37.505699063741375
 pair12 True (-0.00263966, -8.335778e-05) (-0.00269523, -8.335778e-05) ...   0.6741125757416147 41.692103185536034  -40.30476570000266
 pair13 True (-0.05821146, -8.335771e-05)   (-0.05815589, -0.00013893) ...   2.4613671878413927  44.89664184602307  -47.91856854826923
 pair14 True    (-0.03042558, 0.02770256)    (-0.03042558, 0.02764698) ...   3.1608577327658733  41.07589462243317  -36.56538855134798
 pair15 True   (-0.00263966, -0.02786928)   (-0.00269523, -0.02786928) ...   3.9095785158717464  38.13166417802683  -34.02817058690904
 pair16 True   (-0.03042558, -0.02786927)   (-0.03042558, -0.02786927) ... -0.01614557753727189  41.10125358279642  -49.52202476660966
 pair17 True   (-0.05821145, -0.02786925)   (-0.05815588, -0.02786925) ...   1.0207957206965301  42.59523394110048 -42.839296409741536
 pair18 True   (-0.00263966, -0.05565517)   (-0.00269523, -0.05559959) ...    5.901836013265612  45.98010582607783  -38.41669740030114
 pair19 True   (-0.03042556, -0.05565515)   (-0.03042556, -0.05559958) ...   10.110786071826995  33.36011368015603  -32.42363023338399
 pair20 True   (-0.05821142, -0.05565511)   (-0.05815584, -0.05559954) ...    7.748251120039328  36.77703640498484  -26.39317864515166
 pair21 True    (-0.00269523, 0.05548845)    (-0.00269523, 0.05537731) ...    4.640904016777619 35.467498877988206  -35.71493852268121
 pair22 True    (-0.03042556, 0.05548844)    (-0.03042556, 0.05537729) ...   3.9559868574106636  40.52591641143584  -39.30718665348717
 pair23 True     (-0.05821142, 0.0554884)    (-0.05815585, 0.05537726) ...   1.0348728236687468   39.9540878313354  -36.47176481982561
 pair24 True    (-0.00263966, 0.02770256)    (-0.00269523, 0.02764699) ...    6.257958042481342  36.61132670044153  -36.26792424702971
 pair25 True     (0.05287654, 0.05548841)     (0.05282097, 0.05537726) ...  -2.1311369365367043  34.83488722449716  -37.84302682795591

QTables can also hold metadata, so before this gets merged, I'd like to propagate some other items like the camera rotator and boresight pointing. It'd be nice to propagate the flux and blends from donutCat into the rows here too; aggregate everything in one place.

[suberlak]

How about other properties that are currently stored in donutStamps.metadata ? Right now these include
['SIMPLE', 'BITPIX', 'NAXIS', 'EXTEND', 'COMMENT', 'RA_DEG', 'DEC_DEG', 'DET_NAME', 'CAM_NAME', 'VISIT', 'DFC_TYPE', 'DFC_DIST', 'MAG', 'CENT_X0', 'CENT_Y0', 'CENT_X', 'CENT_Y', 'CENT_DX', 'CENT_DY', 'CENT_DR', 'BLEND_CX', 'BLEND_CY', 'X0', 'Y0', 'SN', 'SIGNAL_MEAN', 'SIGNAL_SUM', 'NPX_MASK', 'BKGD_STDEV', 'SQRT_MEAN_VAR', 'BKGD_VAR', 'BACKGROUND_IMAGE_MEAN', 'NOISE_VAR_BKGD', 'NOISE_VAR_DONUT', 'EFFECTIVE', 'ENTROPY', 'PEAK_HEIGHT', 'BANDPASS', 'HAS_MASK', 'HAS_VARIANCE', 'HAS_ARCHIVE', 'N_STAMPS', 'STAMPCLS', 'VERSION', 'ARCHIVE_IDS']

At least a subset of these is very useful, as then we'd have one "all information about each donut pair" table (without the need to utilize multiple tables, especially the getArray method which makes it not very easy to at a glance see all this info for all donuts in the exposure).

I think also throwing in the 4 columns from donutsIntraQuality , donutsExtraQuality would be good. All that would be needed is a prefix eg. intra_sn , extra_sn, intra_mag , extra_mag , etc. That would help with quick selection of eg. which Zernike results we'd like to highlight.

[jmeyers314]

How about other properties that are currently stored in donutStamps.metadata ?

Yep, I'm currently considering which columns may be the most useful. I think

MAG, SN, ENTROPY

is probably sufficient for the main table. The table can also have exposure-level metadata for things like

DET_NAME, CAM_NAME, DFC_DIST

[suberlak]

Yes, some exposure-level metadata are used to populate an entire donut catalog-length array such as
DET_NAME, CAM_NAME, DFC_TYPE, VISIT , DFC_DIST

ts_wep/python/lsst/ts/wep/task/cutOutDonutsBase.py

Lines 680 to 687 in 3dfe1ab

	stampsMetadata["DET_NAME"] = np.array([detectorName] * catalogLength, dtype=str)
	stampsMetadata["CAM_NAME"] = np.array([cameraName] * catalogLength, dtype=str)
	stampsMetadata["VISIT"] = np.array([visitId] * catalogLength, dtype=int)
	stampsMetadata["DFC_TYPE"] = np.array(
	[defocalType.value] * catalogLength, dtype=str
	)
	stampsMetadata["DFC_DIST"] = np.array(
	[instrument.defocalOffset * 1e3] * catalogLength

. They could go to a separate exposure-level table (but then we miss out on the niceness of having just one table with all donut & Zernikes info)

[jmeyers314]

It'd be the same table (same dataset). Just in the metadata, not in the rows.

from astropy.table import Table
table = Table()
table.meta['DET_NAME'] = 'R22_S22'

etc.

[suberlak]

given that sn and entropy information is carried over, did you want to also include the decision flags that get created in donutStampSelector , eg.
SN_SELECT, ENTROPY_SELECT, FINAL_SELECT? Or is it better to keep it separate in donutsIntraQuality, donutsExtraQuality ?

[jmeyers314]

This table only adds the selected donuts, so those columns would be trivially True. So I left them out. Keeping the quality tables, (or maybe adding that info to the donutCat tables?) seems reasonable (for a separate ticket).

[suberlak]

Btw, is the plan to keep outputZernikesRaw,
outputZernikesAvg here, and in the next PR remove them? Once that happens, it would have to be a major version change as it would break backwards compatibility. But in the transition, it's fine to have more outputs than less.

[jmeyers314]

Yep, that was my thought as well. It might not be the very next PR where they're removed, but sometime soon. And at that time we'll bump the major version number.