dataset.tex

\section {Dataset choice considerations} \label{sec:datasetHistory}

{\bf For DP0 we are moving ahead with DESC DR6 as the baseline.}
This section is included for historical context on the decision making.

 There were  two leading candidates for forming the basis of DP0:

\begin{itemize}

\item The Subaru Hyper Suprime-Cam PDR2 dataset, provided permission can be secured from our HSC colleagues. As real (on-sky) data it is likely that users will interact with it in more realistic ways. It is a well understood dataset, and it is regularly re-processed with software that shares a common codebase with the LSST Science Pipelines.

\item The simulated precursor to LSST data produced by the Dark Energy Science Collaboration, DESC DC2, provided permission can be secured. This is a very large dataset and putting DC2 catalogs in Qserv would be an excellent demonstration of its abilities.

\end{itemize}

There is interest from the science collaborations in working with data products from both of these datasets. DC2 was emphasized at the 2019 PCW, and at least one (AGN) has contributed to the simulation inputs since then. A comment at the PCW discussion was that without DC2 in DP0, the science collaborations would not see full frame LSST data until the year before the survey, too late for the needed analysis development.
% PJM: to re-live the PCW2019 session, see the slide deck at https://docs.google.com/presentation/d/1tRHdSyXBRp850zfJO5NssHS-fGfGMMngDEfAD9Azdk4/edit#slide=id.g5e4570c6eb_0_10 and the notes at https://docs.google.com/document/d/1fSNwsT12hTQGZsH--C6sIY58k6ODHx4vnsPfQa82t9Q/edit#heading=h.8o7p552v6klp

Data Management is currently in transition between its 2nd and 3rd generation data abstraction layer (aka ``Butler''). For DP0 to fulfill its aim as an early deployment/integration exercise, Gen 3 Butler must be used, preferably (stretch goal) using an S3 compliant Object Store as is the intent in production. This has bearing on the choice of dataset.

HSC PDR2 can either be converted from Gen 2 to Gen 3 or (stretch goal but ideally) reprocessed naively with Gen3. A smaller subset may be necessary to avoid production scaling issues. This is the preferred choice in the short term from an engineering point of view.

DC2 is available through Gen2 Butler and as we do not process that data with the Science Pipelines, the only option is conversion to Gen3. Estimates are that this is such a time-consuming process that it cannot be done in time to meet milestone L2-DP-0020. Therefore if DC2 is to be involved in the short term, a significantly smaller subset would have to be selected.

In the case that we do not reprocess the data with updated Science Pipelines, we can serve the data as they are provided to us.
For example, in DC2, DESC's codes were used to generate science-ready catalogs, which can be ingested into Qserv without further standardization.
Detailed provenance between images in the Butler repo and the Qserv catalogs may not be provided in DP0.1, but improvements will be made in DP0.2 when we reprocess the data.

Questions:

\begin{itemize}

\item Which dataset has the broader scientific interest? This question could be answered via a community survey: indeed, the possibility of such a survey was discussed at the 2019 PCW.

\item For either dataset if we take a subset to avoid the Gen2-Gen3 conversion issues or production scaling issues, will that reduce the usefulness of the datasets or affect the choice? What would be the smallest data size that is still scientifically interesting?

\item Are there HiPS maps available for either of these ?

%butler moved to risks

\item Given the delayed construction/commissioning schedule, could we consider including both of these datasets in DP0 over the course of FY21--FY22?

\end{itemize}