Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement some flow blocks #22

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
456972e
Split block
evren-okcu Nov 22, 2024
4c9cd5f
merge conflict
evren-okcu Nov 22, 2024
a6b2ca1
wip
evren-okcu Nov 23, 2024
fcbe544
remove public test modules
evren-okcu Nov 23, 2024
0684169
concat fix
evren-okcu Nov 25, 2024
3f9d2cb
wip
evren-okcu Nov 26, 2024
c59b5d5
merge flow
evren-okcu Dec 4, 2024
6d18ff4
comparable any
evren-okcu Dec 4, 2024
4f421b3
Merge branch 'master' into flow-blocks-merge
evren-okcu Dec 6, 2024
418645a
Merge branch 'flow-merge' into flow-blocks-merge
evren-okcu Dec 6, 2024
9f1e8fd
fix readme
evren-okcu Dec 9, 2024
4109572
remove stop port from sort block
evren-okcu Dec 9, 2024
2059ef5
distinct
evren-okcu Dec 9, 2024
446391c
batch block
evren-okcu Dec 10, 2024
6f03218
Merge branch 'master' into flow-distinct-merge
evren-okcu Dec 10, 2024
390cd4b
remove duplicate hash feature configurations
evren-okcu Dec 13, 2024
fbba02f
fix hash feature bug in another branch/PR
evren-okcu Dec 13, 2024
d046bfb
fix Concat it readme
evren-okcu Dec 13, 2024
5b55a23
fix Concat it readme
evren-okcu Dec 13, 2024
ec102fd
compare values instead of types
evren-okcu Dec 13, 2024
cb280b7
serde, std, tracing feature gates
evren-okcu Dec 13, 2024
c5a00e1
Merge branch 'master' into flow-blocks-wip
evren-okcu Dec 13, 2024
0a535d8
distinct merge
evren-okcu Dec 13, 2024
a53a1c5
Merge branch 'flow-batch' into flow-blocks-wip
evren-okcu Dec 13, 2024
419ece7
Merge branch 'hash-feature-fix' into flow-blocks-wip
evren-okcu Dec 13, 2024
74ea51f
merge
evren-okcu Dec 13, 2024
a82f436
tracing feature gate
evren-okcu Dec 13, 2024
8bae9f0
logging
evren-okcu Dec 13, 2024
056a913
logging
evren-okcu Dec 13, 2024
cf7f38b
remove duplicate 'concat' in readme
evren-okcu Dec 16, 2024
65ea358
merge conflicts
evren-okcu Dec 23, 2024
d2f795c
use tracing macros
evren-okcu Dec 23, 2024
f1506ff
fix doc of batch block
evren-okcu Dec 23, 2024
d7df797
MapInto block
evren-okcu Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ The built-in blocks provided by Protoflow are listed below:
|:------------------|:-------------------------------------------------------------------------------------------------------------------------------|
| [`Buffer`] | Stores all messages it receives. |
| [`ConcatStrings`] | Concatenates the received string messages, with an optional delimiter string inserted between each message. |
| [`Concat`] | Merges multiple input message streams into a single output stream. |
| [`Const`] | Sends a constant value. |
| [`Count`] | Counts the number of messages it receives, while optionally passing them through. |
| [`Decode`] | Decodes messages from a byte stream. |
Expand All @@ -133,6 +134,9 @@ The built-in blocks provided by Protoflow are listed below:
| [`ReadFile`] | Reads bytes from the contents of a file. |
| [`ReadSocket`] | Reads bytes from a TCP socket. |
| [`ReadStdin`] | Reads bytes from standard input (aka stdin). |
| [`Replicate`] | Duplicates a single input message stream into multiple identical output streams. |
| [`Sort`] | Sorts a single input message stream in ascending order. |
| [`Split`] | Divides a single input message stream into multiple output streams using a round-robin approach. |
| [`SplitString`] | Splits the received input message, with an optional delimiter string parameter. |
| [`WriteFile`] | Writes or appends bytes to the contents of a file. |
| [`WriteSocket`] | Writes bytes to a TCP socket |
Expand Down Expand Up @@ -618,6 +622,32 @@ block-beta
protoflow execute ReadStdin < input.txt
```

#### [`Split`]

Divides a single input message stream into multiple output streams using a round-robin approach.

```mermaid
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Split space:3
space:5 Sink2 space:1
Source-- "input" -->Split
Split-- "output_1" -->Sink1
Split-- "output_2" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Split block
class Source hidden
class Sink1 hidden
class Sink2 hidden
```

```bash
protoflow execute Split
```

#### [`SplitString`]

A block that splits the received input message, with an optional delimiter string parameter
Expand Down Expand Up @@ -795,6 +825,7 @@ To add a new block type implementation, make sure to examine and amend:
[`examples`]: lib/protoflow/examples

[`Buffer`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Buffer.html
[`Concat`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Concat.html
[`ConcatStrings`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ConcatStrings.html
[`Const`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Const.html
[`Count`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Count.html
Expand All @@ -815,6 +846,9 @@ To add a new block type implementation, make sure to examine and amend:
[`ReadFile`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadFile.html
[`ReadSocket`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadSocket.html
[`ReadStdin`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadStdin.html
[`Replicate`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Replicate.html
[`Sort`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Sort.html
[`Split`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Split.html
[`SplitString`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.SplitString.html
[`WriteFile`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.WriteFile.html
[`WriteSocket`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.WriteSocket.html
Expand Down
15 changes: 15 additions & 0 deletions lib/protoflow-blocks/doc/flow/split.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Split space:3
space:5 Sink2 space:1
Source-- "input" -->Split
Split-- "output_1" -->Sink1
Split-- "output_2" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Split block
class Source hidden
class Sink1 hidden
class Sink2 hidden
29 changes: 29 additions & 0 deletions lib/protoflow-blocks/doc/flow/split.seq.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
sequenceDiagram
autonumber
participant BlockA as Another block
participant Split.input as Split.input port
participant Split as Split block
participant Split.output_1 as Split.output_1 port
participant BlockB as Another block
participant Split.output_2 as Split.output_2 port
participant BlockC as Another block

BlockA-->>Split: Connect
Split-->>BlockB: Connect
Split-->>BlockC: Connect

loop Split process
BlockA->>Split: Message
Split->>Split: Decide the next output port
Split->>BlockB: Message
BlockA->>Split: Message
Split->>Split: Decide the next output port
Split->>BlockC: Message
end

BlockA-->>Split: Disconnect
Split-->>Split.input: Close
Split-->>Split.output_1: Close
Split-->>BlockB: Disconnect
Split-->>Split.output_2: Close
Split-->>BlockC: Disconnect
12 changes: 6 additions & 6 deletions lib/protoflow-blocks/src/block_config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,6 @@ pub enum BlockConfig {
feature = "hash-sha1",
feature = "hash-sha2"
))]
#[cfg(any(
feature = "hash-blake3",
feature = "hash-md5",
feature = "hash-sha1",
feature = "hash-sha2"
))]
SamuelSarle marked this conversation as resolved.
Show resolved Hide resolved
Hash(HashBlockConfig),
Io(IoBlockConfig),
Math(MathBlockConfig),
Expand Down Expand Up @@ -63,6 +57,12 @@ impl<'de> serde::Deserialize<'de> for BlockConfig {
.unwrap()
}

"Concat" | "Replicate" | "Split" | "Sort" => {
FlowBlockConfig::deserialize(value.clone())
.map(BlockConfig::Flow)
.unwrap()
}

#[cfg(any(
feature = "hash-blake3",
feature = "hash-md5",
Expand Down
16 changes: 16 additions & 0 deletions lib/protoflow-blocks/src/block_tag.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ pub enum BlockTag {
Drop,
Random,
// FlowBlocks
Concat,
Replicate,
Sort,
Split,
// HashBlocks
#[cfg(any(
feature = "hash-blake3",
Expand Down Expand Up @@ -73,6 +77,7 @@ impl BlockTag {
use BlockTag::*;
match self {
Buffer => "Buffer",
Concat => "Concat",
Const => "Const",
Count => "Count",
Delay => "Delay",
Expand Down Expand Up @@ -101,6 +106,9 @@ impl BlockTag {
ReadSocket => "ReadSocket",
#[cfg(feature = "std")]
ReadStdin => "ReadStdin",
Replicate => "Replicate",
Sort => "Sort",
Split => "Split",
#[cfg(feature = "std")]
WriteFile => "WriteFile",
#[cfg(feature = "std")]
Expand All @@ -124,6 +132,7 @@ impl FromStr for BlockTag {
use BlockTag::*;
Ok(match input {
"Buffer" => Buffer,
"Concat" => Concat,
"Const" => Const,
"Count" => Count,
"Delay" => Delay,
Expand Down Expand Up @@ -152,6 +161,9 @@ impl FromStr for BlockTag {
"ReadSocket" => ReadSocket,
#[cfg(feature = "std")]
"ReadStdin" => ReadStdin,
"Replicate" => Replicate,
"Sort" => Sort,
"Split" => Split,
#[cfg(feature = "std")]
"WriteFile" => WriteFile,
#[cfg(feature = "std")]
Expand Down Expand Up @@ -186,6 +198,7 @@ impl BlockInstantiation for BlockTag {
use BlockTag::*;
match self {
Buffer => Box::new(super::Buffer::<Any>::with_system(system)),
Concat => Box::new(super::Concat::<Any>::with_system(system)),
Const => Box::new(super::Const::<String>::with_system(system, String::new())),
Count => Box::new(super::Count::<Any>::with_system(system)),
Delay => Box::new(super::Delay::<Any>::with_system(system, None)),
Expand Down Expand Up @@ -214,6 +227,9 @@ impl BlockInstantiation for BlockTag {
ReadSocket => Box::new(super::ReadSocket::with_system(system, None)),
#[cfg(feature = "std")]
ReadStdin => Box::new(super::ReadStdin::with_system(system, None)),
Replicate => Box::new(super::Replicate::<Any>::with_system(system)),
Sort => Box::new(super::Sort::<Any>::with_system(system)),
evren-okcu marked this conversation as resolved.
Show resolved Hide resolved
Split => Box::new(super::Split::<Any>::with_system(system)),
#[cfg(feature = "std")]
WriteFile => Box::new(super::WriteFile::with_system(system, None)),
#[cfg(feature = "std")]
Expand Down
124 changes: 116 additions & 8 deletions lib/protoflow-blocks/src/blocks/flow.rs
Original file line number Diff line number Diff line change
@@ -1,30 +1,138 @@
// This is free and unencumbered software released into the public domain.

pub mod flow {
use crate::{InputPortName, OutputPortName};

use super::{
prelude::{Cow, Named},
BlockConnections, BlockInstantiation,
prelude::{vec, Box, Cow, Named, Vec},
BlockConnections, BlockInstantiation, System,
};

pub trait FlowBlocks {}
use protoflow_core::{Block, Message};

pub trait FlowBlocks {
fn concat<T: Message + Into<T> + 'static>(&mut self) -> Concat<T>;
fn replicate<T: Message + Into<T> + 'static>(&mut self) -> Replicate<T>;
fn sort<T: Message + Into<T> + PartialOrd + 'static>(&mut self) -> Sort<T>;
fn split<T: Message + Into<T> + 'static>(&mut self) -> Split<T>;
}

#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub enum FlowBlockTag {}
pub enum FlowBlockTag {
Concat,
Replicate,
Sort,
Split,
}

#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[derive(Clone, Debug)]
pub enum FlowBlockConfig {}
pub enum FlowBlockConfig {
Concat {
input_1: InputPortName,
input_2: InputPortName,
output: OutputPortName,
},
Replicate {
input: InputPortName,
output_1: OutputPortName,
output_2: OutputPortName,
},
Sort {
input: InputPortName,
stop: InputPortName,
output: OutputPortName,
},
Split {
input: InputPortName,
output_1: OutputPortName,
output_2: OutputPortName,
},
}

impl Named for FlowBlockConfig {
fn name(&self) -> Cow<str> {
unreachable!()
use FlowBlockConfig::*;
Cow::Borrowed(match self {
Concat { .. } => "Concat",
Replicate { .. } => "Replicate",
Sort { .. } => "Sort",
Split { .. } => "Split",
})
}
}

impl BlockConnections for FlowBlockConfig {}
impl BlockConnections for FlowBlockConfig {
fn output_connections(&self) -> Vec<(&'static str, Option<OutputPortName>)> {
use FlowBlockConfig::*;
match self {
Concat { output, .. } => {
vec![("output", Some(output.clone()))]
}
Replicate {
output_1, output_2, ..
} => {
vec![
("output_1", Some(output_1.clone())),
("output_2", Some(output_2.clone())),
]
}
Sort { output, .. } => {
vec![("output", Some(output.clone()))]
}
Split {
output_1, output_2, ..
} => {
vec![
("output_1", Some(output_1.clone())),
("output_2", Some(output_2.clone())),
]
}
}
}
}

impl BlockInstantiation for FlowBlockConfig {
fn instantiate(&self, system: &mut System) -> Box<dyn Block> {
use super::SystemBuilding;
use FlowBlockConfig::*;
match self {
Concat { .. } => Box::new(super::Concat::new(
system.input_any(),
system.input_any(),
system.output(),
)),
Replicate { .. } => Box::new(super::Replicate::new(
system.input_any(),
system.output(),
system.output(),
)),
Sort { .. } => Box::new(super::Sort::new(
system.input_any(),
SamuelSarle marked this conversation as resolved.
Show resolved Hide resolved
system.input(),
system.output(),
)),
evren-okcu marked this conversation as resolved.
Show resolved Hide resolved
Split { .. } => Box::new(super::Split::new(
system.input_any(),
system.output(),
system.output(),
)),
}
}
}

mod concat;
pub use concat::*;

mod replicate;
pub use replicate::*;

mod sort;
pub use sort::*;

impl BlockInstantiation for FlowBlockConfig {}
mod split;
pub use split::*;
}

pub use flow::*;
Loading