-
Notifications
You must be signed in to change notification settings - Fork 3
/
plan.tex
119 lines (81 loc) · 7.6 KB
/
plan.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
\section{Planning and team(s) fro DP0} \label{sec:plan}
Planning epics have been (and are) being created in the \href{https://jira.lsstcorp.org/secure/Dashboard.jspa?selectPageId=15608}{PREOPS Jira} project.
On the dashboard you can see links to the tickets labeled DP0.1 and DP0.2.
We will have regular (every other week for now) DP0 meetings (see \url{https://confluence.lsstcorp.org/display/LSSTOps/Data+Production+Meetings}).
\subsection {Teams}
The Operations era org chart is shown in \figref{fig:org}.
\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{images/Ops_Org_Chart}
\end{center}
\caption{ Organization of departments and teams for operations of Rubin Observatory. \label{fig:org}}
\end{figure}
The main departments involved in DP0 are Data Production and System Performance. With in those departments various people will be involved from the underlying teams but in small numbers. It makes most sense to approach DP0 with a task force approach. This might best be seen as two teams:
\begin{itemize}
\item Data production - with a focus on middleware and execution (\secref{sec:dp});
\item System Performance - with a focus on quality assurance and community support (\secref{sec:sp}).
\end{itemize}
As we advance the teams grow and we will transition to the an organization as in \figref{fig:org}
with team leads for each team as in \figref{fig:dporg}.
\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{images/DpOrg}
\end{center}
\caption{Data Production group/team structure \label{fig:dporg}}
\end{figure}
For a more detailed description of infrastructure see \citeds{RTN-021}.
\subsubsection{Task force lead}\label{sec:tfl}
For DP0 on the IDF a task force approach seems most appropriate given the partial efforts in all teams.
Hsin-Fang Chiang shall fulfill this role and coordinate Data Production activities for DP0.
Responsibilities of this role include:
\begin{itemize}
\item Being point of contact for the IDF provider.
\item Setting priorities for all work at the IDF until DP0 is fully complete.
\item Evaluate stage-wise operational readiness wrt. to requirements.
\item Make all components of the IDF work together (Science Platform, Middleware, Workflow ..)
\end{itemize}
The task force lead reports directly to the Data Production Associate Director and
carries delegated authority for the above responsibilities.
\subsection{DP Middleware and Execution}\label{sec:dp}
There is preops effort (fractional FTE) available in Execution and Pipelines as well as Middleware teams.
The roles etc need some clean up from the ops proposal but the DP Roles are listed in \tabref{tab:teamsNames} though the exact mix of roles is still under discussion.
\subsection{SP Quality and Community Support} \label{sec:sp}
Note: DP0.1 and DP0.2 Early Access described in this document do not leave time for full-scale quality analysis. The provided data will not be science-ready; system performance milestones are succeeding.
Besides extensive documentation at \href{https://dp0-1.lsst.io}{dp0-1.lsst.io} supporting delegates in a number of ways, delegate resources included biweekly virtual ``Delegate Assemblies'' with hands-on demonstrations and breakout rooms to facilitate co-working; a dedicated category in the Rubin Community Forum (\url{https://community.lsst.org}) (based on the Discourse web forum platform) where delegates can ask questions and discuss their DP0 work; and a GitHub repository where delegates can contribute to shared code and notebooks.
GitHub Issues were also used as the primary means for delegates to request technical assistance. All of the delegate pedagogical resources are publicly accessible.
Some aspects of the Community Engagement model as applied to DP0 are designed to scale to thousands of users for the future Data Previews, such as the extensive documentation, the tutorials, and the use of the Community Forum.
Other aspects, such as the frequent live virtual sessions, may not scale well to thousands of users and were specifically designed to ramp up skill levels from novice to intermediate in order to seed expertise with Rubin software in the community.
For future Data Previews, the Community Engagement strategy will focus on building infrastructure to foster a vibrant science community, that enables self-help, peer-to-peer help and crowd sourced support.
\subsection{Planning}
\input{tables/timeline}
Table \ref{tab:timeline} lists internal timeline.
\subsubsection{Middleware}
There are obvious middleware milestones such as L3-MW-0030 read only Gen3 Butler which are needed from the construction project.
There is still installation work needed for the that on Google which includes the need for a Postgress (like) database for the registry. The DAX team are on the hook for this.
For DP0.2 we need Butler to handle processing, not just locating files (L3-MW-0070).
\paragraph{Qserv} should be installed and configured. Though we have some prior art for this we
still will need some experimentation to get it correct. Getting DC2 loaded in Qserv is also a DAX activity
we will have to do on IDF.
\paragraph{Workflow} needs to be functioning at scale for DP0.2, ideally we should have basic workflow
early on (milestone L3-MW-0050). Then more tooling such as restarting failed jobs (L3-MW-0060).
From the construction side we have BPS as a deliverable which may be useful on IDF also.
We shall evaluate BPS as an option later in 2020 (L3-MW-0040).
See \citeds{LDM-636}, \citeds{LDM-633}, \citeds{DMTN-123}.
BPS translates the quantum graph to DAGMan for execution on HTCondor and submits the jobs.
Most work has gone into the graph and execution.
As part of our march toward a potential more DOE oriented Data Facility, BNL will be part of the pre operations team to experiment with PanDA as an environment to monitor and control our processing jobs.
This is a slightly parallel effort to construction attempting to take advantage of an existing set of tools for large scale job execution.
In an ideal world the quantum graph translation of BPS would feed into a PanDA system to execute (retry etc) our jobs, this is still to be investigated. This may go through CWL.
See also \secref{sec:in2p3}.
\subsubsection{ Science Platform}
The science platform and web services need to be deployed. In principle this is reasonable straight forward, an open issue may be configuring of the Portal aspect for the chosen dataset(s).
\subsubsection{ Pipelines }
For DP0.2 we need a Gen3 version of the pipelines to process the dataset. This will have to run at scale for PDR2 or DC2. There may be several runs for quality purposes.
Fractional FTE from the Pipelines will provide help in pipeline configuration, data repo preparation, workflow consulting, science verification, data model documenting, troubleshooting, and liaising.
{\bf Yusra will provide more info here.}
\subsubsection{ IN2P3}\label{sec:in2p3}
IN2P3 will contribute in Qserv and pipelines. {\bf Fabio will provide more information here.}
They bring experience running Gen3 workflows. The real interest with IN2P3 is to run remote jobs thus emulating the eventual operational DRP runs. This may be difficult to achieve in FY21 but we should make it a milestone for FY22.\footnote{Tim, Fabio we should set a date for this}. A more achievable goal for FY21 would be to duplicate the IDF processing at IN2P3.
Remote execution requires some features in Gen3 to be implemented. We will probably wish to execute jobs with a local registry then merge the results and registries.
IN2P3 maintains a separate Qserv cluster.
The same catalog data will be ingested into the Qserv instance at IDF and the Qserv instance at IN2P3 and the databases will be cross-checked for consistency.