Status Update: 2022-09-14

[phil-opp]

In this discussion, we describe our recent progress and the main topics that we would like to discuss. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[phil-opp]

New communication-layer crate

• crate is independent of dora, so it can be reused in other projects (e.g. in essa)
• New communication layer design with separate creation step for publishers/subscribers
• All methods of the API are synchronous now. For example, the inputs method now returns a channel instead of an asynchronous stream

Detailed Interface:

/// Abstraction trait for different publisher/subscriber implementations.
pub trait CommunicationLayer: Send + Sync {
    /// Creates a publisher for the given topic.
    fn publisher(&mut self, topic: &str) -> Result<Box<dyn Publisher>, BoxError>;

    /// Subscribe to the given topic.
    fn subscribe(&mut self, topic: &str) -> Result<Box<dyn Subscriber>, BoxError>;
}

/// Allows publishing messages to subscribers.
pub trait Publisher: Send + Sync {
    /// Publish the given data to subscribers.
    ///
    /// The data is published to the topic that was used to create this publisher
    /// (see [`CommunicationLayer::publisher`]).
    fn publish(&self, data: &[u8]) -> Result<(), BoxError>;

    /// Clone this publisher, returning the clone as a
    /// [trait object](https://doc.rust-lang.org/book/ch17-02-trait-objects.html).
    fn dyn_clone(&self) -> Box<dyn Publisher>;
}

/// Allows receiving messages published on a topic.
pub trait Subscriber: Send + Sync {
    /// Receives the next message.
    ///
    /// Blocks until the next message is available.
    ///
    /// Depending on the chosen communication backend, some messages might be dropped if
    /// the publisher is faster than the subscriber.
    fn recv(&mut self) -> Result<Option<Vec<u8>>, BoxError>;
}

Metadata serialization

We now use a higher-level message format that is serialized using capnproto. This way, we can send additional metadata with messages, for example OpenTelemetry contexts.

[phil-opp]

Iceoryx

In addition to zenoh, the communication layer now also supports communication using iceoryx. Iceoryx has the following properties:

• pub/sub: Like zenoh, iceoryx uses a publish/subscribe pattern.
• shared memory: Instead of communicating via sockets or pipes, iceoryx uses shared memory.
• local only: Iceoryx is only able to send messages to nodes/operators running on the same machine.
• Roudi daemon: The allocation of shared memory is done by a central daemon called RouDi. This daemon must be started first in order to use iceoryx as communication layer.
• zero-copy: Since the central RouDi daemon is in control of all shared memory, it is possible to send a message to multiple subscribers without copying it.

While we haven't run any benchmarks yet, we expect that communication over iceoryx should be much faster than with zenoh. We are currently working on using iceoryx for dora-drives, which should give us a performance comparison for a real-world use case.

[phil-opp]

Command line interface

We want to create a dora command-line tool for managing dataflow runs. It should also provide helper functions for creating dataflows and for viewing logs and statistics. Our initial draft looks like this:

Usage:

dora [flags] <command> [<args>]

Flags:
  -h, --help    Print this help
  -v, --version Print version info (cli, control plane, data plane)

Available commands:
  help         Help about any command

  check        Pre-check the dora installation/environment for potential problems (e.g., YAML syntax)
  graph        Generate dataflow graph per YAML spec
  build        Build the dora nodes/operators (similar to run.rs in current examples)
  
  templates    Dora node/operator template

  up           Install all dora dependencies and start dora coordinator, iceoryx daemon and/or zenoh router etc
  start        Run dora dataflow
  stop         Stop dora dataflow
  destroy      Cleanup and destroy all dora related processes
  
  dashboard    Open the dora dashboard in a web browser (future)
  logs         Tail logs from customer node/operators
  metrics      Fetch metrics from customer node/operators
  stats        Display dora/traffic statistics 
  
  list         List running dora entities (resources)
  get          Get info about the dora entity (future may need CRUD on dora entity)
  
  upgrade      Upgrade dora 

[heyong4725]

In the future, we may develop dora external APIs such that it can be integrated with other tools/platform. The APIs will provide equivalent features as dora CLI interface (which is human user interface) to interface with other computer systems. The tool https://github.com/OpenAPITools/openapi-generator may provide some help

[heyong4725]

When develop Dora CLI, we also envision that we will develop Dora Hub (similar to Docker Hub) which will host Dora modules (e.g., python module, shared library, WASM module, executables) with their associated metadata (e.g., version, signature...). Docker Hub and Docker CLI provide us good reference

[heyong4725]

Regarding to how to package Dora modules, this open source project https://github.com/deislabs/bindle offers very good solution. We will investigate if this is the right tool for Dora.

[phil-opp]

Future Work

• Zero-copy communication: Update the communication layer to support zero-copy messaging with iceoryx. For this, we would need an additional prepare_message(size: usize) method, which allocates the required shared memory buffer through the iceoryx RouDi daemon. Nodes/operators can then construct messages directly into that shared memory buffer, which can then be shared with other nodes/operators.
• Log and Replay
  • option to turn on message logging on all nodes/operators (might impact performance)
  • acquired message logs can then be used to replay execution on a virtual system
  • useful for debugging
• Communicate over mpsc channels: Use in-memory channels for communication within a runtime node when it runs multiple operators. This allows us to skip the serialization, which should result in further performance improvements compared with shared memory.
• Tiered communication layers: Support multiple communication layers in runtime and node APIs. Depending on the target of a message, the fastest layer would be used. For example, a message would be sent using iceoryx if the target runs on the same machine and using zenoh if it is remote.

See also our updated roadmap.

[heyong4725]

@phil-opp We may expose prepare_message(size: usize) method as part of Dora SDK API, such that let Dora core to manage memory/buffers, in this way, Python module may be able to leverage it to gain a little performance?