Feature: WorkcellManager

[LuckierDodge]

Add support for a workcell manager to MADSci, to handle scheduling and executing workflows across multiple related nodes. This will port much of the core functionality from the WEI Workcell Engine.

The WorkcellManager will consist of a madsci.workcell_manager which acts as the single controller and manager for all workflows, resources, and data related to a particular workcell. It communicates directly with it's nodes via appropriate NodeClients, with the database(s) via their appropriate clients or ORMs, and with other managers as needed to conduct tasks related to the workcell.

It will also include a standardized workcell_client in the madsci.clients package, which will allow users and other systems to programmatically interact with the workcell.

Functional Requirements

• The system must accept new workflow submissions, adding them to a workflow queue
• The system shall accept workflow sequences, to be run in an explicit order
• The system shall accept workflow batches, to be run in any order
• The system shall accept reservation requests, to reserve part or all of the workcell exclusively for the purposes of a workflow, experiment, campaign, user, project, etc; these reservations must be time limited.
• The system must validate each submitted workflow against the Workflow type
• The system must validate each step in a workflow against the target node's NodeInfo, if available
• The system must accept file uploads alongside new Workflows, to be used as file arguments in individual steps
• The system must read from a Workcell Definition file on startup (*.workcell.yaml), and validate that definition against the WorkcellDefinition type
• The system must support updating the workcell definition via the client, and should allow both ephemeral changes and changes that update the definition file
• The system must accept Workflows with Parameters, and should be able to substitute those parameters prior to running the workflow
• The system shall support querying the system for the state of nodes managed by the workcell
• The system shall support querying the system for the state of resources managed by the workcell
• The system shall support querying the system for the state of workflow runs managed by the workcell
• The system shall support sending actions to nodes based on the steps of a workflow
• The system shall support awaiting results from actions sent to nodes, querying the node periodically if the node returns a non-terminal ActionStatus, or if the send_action connection fails before return; this behavior shall be invoked only if the node supports the get_action_result capability
• The system shall include a Scheduler submodule, which will manage when to send actions to the nodes.
• The system shall support a standardized SchedulerMetadata for workflows, where a workflow can be annotated by different scheduler components. Various scheduler components can then annotate the workflow for boolean readiness and float priority, and the scheduler algorithm can use these values to determine the next action to run.
• The system shall allow workcell maintainers to select which scheduler components to enable, and what scheduler algorithm(s) to use (FIFO, priority queuing, etc).
• The system shall support administrative commands (pause, resume, safety stop, cancel, lock, unlock, etc), passing these commands along to nodes as needed/able.
• The system shall support resuming/retrying failed workflows, either from the failed step, from a prior step, or from the beginning of the workflow.
• The system shall support resubmitting a previous workflow as a new workflow (with new workflow_id).