Status Update: 2022-04-27

[haixuanTao]

In this discussion, we describe our recent progress and the main topics that we would like to discuss in today's meeting. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[haixuanTao]

Design Changes

We decided on a new naming convention for light-weight and heavy-weight operators. We also updated our design diagram and the dataflow configuration format to include both operator types.

New Names

• operator instead of light-weight operator
• node instead of heavy-weight operator
  • dora runtime nodes execute operators
  • custom nodes are user-provided executables

In our design documentation, we describe the differences between operators and custom nodes in the following way:

Operators have the following advantages:

• They can use a wide range of advanced functionality provided by the dora runtime nodes. This includes special scheduling strategies and features such as deadlines.
• They are light-weight, so they only occupy minimal amounts of memory. This makes it possible to run thousands of operators on the same machine.
• They can use runtime-managed state storage, for robustness or for sharing state with other operators.
• They share the address space with other operators on the same node, which makes communication much faster.

Custom nodes provide a different set of advantages:

• Each node is a separate, isolated process, which can be important for security-critical operations.
• They support pinned resources. For example, a CPU core can be pinned to a custom node through the dataflow configuration file.
• They have full control over their execution, which allows to create complex input and wake-up rules.

Special Handling for Python Operators

Since Python still uses a global interpreter lock, which effectively limits the possible concurrency across threads, we need some special handling for operators written in Python. Instead of running multiple Python operators on the same node, we spawn a separate runtime subprocess for them. The runtime subprocesses communicate with the parent process via shared memory, so the overhead should be negligible.

Updated Design Diagram

Changes to Dataflow Config Format

We updated the YAML format for specifying nodes and operators.

Nodes:

- id: some-unique-id
  name: Human-Readable Node Name
  description: An optional description of the node's purpose.

  # EITHER:
  operators:
    - id: operator-1
      # ... (see below)
    - id: operator-2
      # ... (see below)

  # OR:
  custom:    
    run: path/to/timestamp
    env:
      - ENVIRONMENT_VARIABLE_1: true
    working-directory: some/path

    inputs:
      input_1: operator_2/output_4
      input_2: custom_node_2/output_4
    outputs:
      - output_1

Operators:

- id: unique-operator-id
  name: Human-Readable Operator Name
  description: An optional description of the operators's purpose.

  inputs:
    input_1: source_operator_2/output_1
    input_2: custom_node_1/output_1
  outputs:
    - output_1
  
  ## ONE OF:
  shared_library: "path/to/shared_lib.so"
  python: "path/to/python_file.py"
  wasm: "path/to/wasm_file.wasm"

New Visualization

We updated our dataflow visualization code to take runtime nodes and operators into account:

flowchart TB
  timer[\timer/]
  camera[\camera/]
subgraph node-1[__node1]
  node-1/operator-1[operator-1]
  node-1/operator-2[operator-2]
end
  printer[/printer\]
  camera -- image --> node-1/operator-1
  node-1/operator-1 -- processed --> node-1/operator-2
  timer -- time --> node-1/operator-2
  node-1/operator-2 -- processed --> printer
  timer -- time --> printer

Loading

[haixuanTao]

State Management and KV Store

We discussed state management and potential access to key-value stores a bit. Our current plan is the following:

• Operators should support different kinds of state.
  • local state: stored internally through a context pointer (or an object instance), not accessible by dora, useful for storing transient state between calls
  • recovery state: key-value store provided by dora, used to recreate crashed operators
  • shared state: eventually-consistent key-value store provided by dora, can be attached to multiple operators (and custom nodes?) through dataflow config file
• Custom nodes need to implement custom state handling.
  • no support for recovery state
  • perhaps access to shared state (if possible)

Decision: Postpone implementation of recovery and shared state for now, focus on other things first.

[phil-opp]

Open questions raised in the meeting regarding shared storage:

• What kinds of storage do we want to support for shared state?

  We should probably support multiple backends, e.g. anna, redis, or relational databases. This way, users can choose the backend that fits their use case best.

• Does shared state only support eventual consistency?

  Since the consistency requirements are application-specific, we should make the storage backend and thus the consistency guarantees configurable, ideally on a per-operator basis.

• Will it be possible to combine different storage backends for shared state in the same dataflow?

  This depends on the API design that we choose, but it should be possible to implement it that way if we want to support that use case. For example, the functions for reading/writing shared memory could take an additional storage_id parameter.

[haixuanTao]

Implementation Progress

We made good progress on both Pylot and the Rust runtime in the past two weeks.

Pylot Advancement

Working E2E pipeline! 🚀 The graph flow look as follows:

flowchart TB
  carla_source_node[\carla_source_node/]
  perfect_detection_node
  obstacles_location_node
  planning_node
  pid_control_node
  control_node
  influxdb_node[/influxdb_node\]
  carla_source_node -- depth_frame --> perfect_detection_node
  carla_source_node -- position --> perfect_detection_node
  carla_source_node -- segmented_frame --> perfect_detection_node
  carla_source_node -- depth_frame --> obstacles_location_node
  perfect_detection_node -- obstacles_without_location --> obstacles_location_node
  carla_source_node -- position --> obstacles_location_node
  obstacles_location_node -- obstacles --> planning_node
  carla_source_node -- position --> planning_node
  carla_source_node -- position --> pid_control_node
  planning_node -- waypoints --> pid_control_node
  pid_control_node -- control --> control_node
  carla_source_node -- vehicle_id --> control_node
  control_node -- control_status --> influxdb_node
  obstacles_location_node -- obstacles --> influxdb_node
  carla_source_node -- position --> influxdb_node

Loading

Added

• Planning operator with RRTSarPlanning, HybridAStarPlanning, and FOTPlanning.
• PID Controller Operator.
• Carla Control Operator.
• Added Timestamp to Messages as Python Code as POC.
• Evaluation metrics of latency for networking and computation.
• InfluxDB connection to visualize the metrics.

Changed

• Better task queueing and task dropping to avoid CPU-starvation.

Fixed

• Pylot-Docker Image issues. Problems was that the Planning Operators used Illegal Instruction from C-Code Compilation. Building the Image from scratch fixed the issue. Build time is considerably longer although caching alleviate this problem.
• Several thread-unsafe code within Pylot operators.

Runtime

• We started to create a general runtime node to run operators. It is already able to communicate with operators on other nodes and with custom nodes in both directions.
• The inital goal is to support shared libraries through a C-compatible API. This way, we support operators written in Rust or C. To make the creation of Rust operators easier, we plan to create a dora-operator-shared-library crate that abstracts away the FFI glue code.
• The runtime uses simple, event-based actor design based on the tokio crate. Each operator is its own actor, either implemented as an asynchronous task or an OS-level thread, depending on the language.
  • In the first iteration, we will focus on synchronous operators implemented as separate threads.
  • Each operators has a separate queue for inputs, to avoid blocking the main actor, which runs the event loop.

[haixuanTao]

Meeting Proposition: Integrate OpenTelemetry for monitoring.

[heyong4725]

During the meeting, we also discussed about Erlang elixir actor model. There is an interesting Rust project https://github.com/lunatic-solutions/lunatic, which is inspired by Erlang actor model, and also has nice support of Wasm.

Please take a look @phil-opp