Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement some flow blocks #22

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
456972e
Split block
evren-okcu Nov 22, 2024
4c9cd5f
merge conflict
evren-okcu Nov 22, 2024
a6b2ca1
wip
evren-okcu Nov 23, 2024
fcbe544
remove public test modules
evren-okcu Nov 23, 2024
0684169
concat fix
evren-okcu Nov 25, 2024
3f9d2cb
wip
evren-okcu Nov 26, 2024
c59b5d5
merge flow
evren-okcu Dec 4, 2024
6d18ff4
comparable any
evren-okcu Dec 4, 2024
4f421b3
Merge branch 'master' into flow-blocks-merge
evren-okcu Dec 6, 2024
418645a
Merge branch 'flow-merge' into flow-blocks-merge
evren-okcu Dec 6, 2024
9f1e8fd
fix readme
evren-okcu Dec 9, 2024
4109572
remove stop port from sort block
evren-okcu Dec 9, 2024
2059ef5
distinct
evren-okcu Dec 9, 2024
446391c
batch block
evren-okcu Dec 10, 2024
6f03218
Merge branch 'master' into flow-distinct-merge
evren-okcu Dec 10, 2024
390cd4b
remove duplicate hash feature configurations
evren-okcu Dec 13, 2024
fbba02f
fix hash feature bug in another branch/PR
evren-okcu Dec 13, 2024
d046bfb
fix Concat it readme
evren-okcu Dec 13, 2024
5b55a23
fix Concat it readme
evren-okcu Dec 13, 2024
ec102fd
compare values instead of types
evren-okcu Dec 13, 2024
cb280b7
serde, std, tracing feature gates
evren-okcu Dec 13, 2024
c5a00e1
Merge branch 'master' into flow-blocks-wip
evren-okcu Dec 13, 2024
0a535d8
distinct merge
evren-okcu Dec 13, 2024
a53a1c5
Merge branch 'flow-batch' into flow-blocks-wip
evren-okcu Dec 13, 2024
419ece7
Merge branch 'hash-feature-fix' into flow-blocks-wip
evren-okcu Dec 13, 2024
74ea51f
merge
evren-okcu Dec 13, 2024
a82f436
tracing feature gate
evren-okcu Dec 13, 2024
8bae9f0
logging
evren-okcu Dec 13, 2024
056a913
logging
evren-okcu Dec 13, 2024
cf7f38b
remove duplicate 'concat' in readme
evren-okcu Dec 16, 2024
65ea358
merge conflicts
evren-okcu Dec 23, 2024
d2f795c
use tracing macros
evren-okcu Dec 23, 2024
f1506ff
fix doc of batch block
evren-okcu Dec 23, 2024
d7df797
MapInto block
evren-okcu Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 139 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ The built-in blocks provided by Protoflow are listed below:
|:------------------|:-------------------------------------------------------------------------------------------------------------------------------|
| [`Buffer`] | Stores all messages it receives. |
| [`ConcatStrings`] | Concatenates the received string messages, with an optional delimiter string inserted between each message. |
| [`Concat`] | Merges multiple input message streams into a single output stream. |
evren-okcu marked this conversation as resolved.
Show resolved Hide resolved
| [`Const`] | Sends a constant value. |
| [`Count`] | Counts the number of messages it receives, while optionally passing them through. |
| [`Decode`] | Decodes messages from a byte stream. |
Expand All @@ -127,12 +128,16 @@ The built-in blocks provided by Protoflow are listed below:
| [`EncodeHex`] | Encodes a byte stream into hexadecimal form. |
| [`EncodeJSON`] | Encodes messages into JSON format. |
| [`Hash`] | Computes the cryptographic hash of a byte stream. |
| [`Merge`] | Combines multiple input message streams into a single output stream by interleaving messages as they arrive. |
| [`Random`] | Generates and sends a random value. |
| [`ReadDir`] | Reads file names from a file system directory. |
| [`ReadEnv`] | Reads the value of an environment variable. |
| [`ReadFile`] | Reads bytes from the contents of a file. |
| [`ReadSocket`] | Reads bytes from a TCP socket. |
| [`ReadStdin`] | Reads bytes from standard input (aka stdin). |
| [`Replicate`] | Duplicates a single input message stream into multiple identical output streams. |
| [`Sort`] | Sorts a single input message stream in ascending order. |
| [`Split`] | Divides a single input message stream into multiple output streams using a round-robin approach. |
| [`SplitString`] | Splits the received input message, with an optional delimiter string parameter. |
| [`WriteFile`] | Writes or appends bytes to the contents of a file. |
| [`WriteSocket`] | Writes bytes to a TCP socket |
Expand All @@ -159,6 +164,32 @@ block-beta
protoflow execute Buffer
```

#### [`Concat`]

Concatenates multiple input message streams into a single output stream.

```mermaid
block-beta
columns 7
space:1 Source1 space:5
space:3 Concat space:1 Sink space:1
space:1 Source2 space:5
Source1-- "input" -->Concat
Source2-- "input" -->Concat
Concat-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Concat block
class Source1 hidden
class Source2 hidden
class Sink hidden
```

```bash
protoflow execute Concat
```

#### [`ConcatStrings`]

A block for concatenating all string messages it receives, with an optional delimiter string inserted between each message
Expand Down Expand Up @@ -483,6 +514,32 @@ block-beta
protoflow execute Hash algorithm=blake3
```

#### [`Merge`]

Combines multiple input message streams into a single output stream by interleaving messages as they arrive.

```mermaid
block-beta
columns 7
space:1 Source1 space:5
space:3 Merge space:1 Sink space:1
space:1 Source2 space:5
Source1-- "input" -->Merge
Source2-- "input" -->Merge
Merge-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Merge block
class Source1 hidden
class Source2 hidden
class Sink hidden
```

```bash
protoflow execute Merge
```

#### [`Random`]

A block for generating and sending a random value.
Expand Down Expand Up @@ -618,6 +675,83 @@ block-beta
protoflow execute ReadStdin < input.txt
```

#### [`Replicate`]

Duplicates a single input message stream into multiple identical output streams.

```mermaid
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Replicate space:3
space:5 Sink2 space:1

Source-- "input" -->Replicate
Replicate-- "output" -->Sink1
Replicate-- "output" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Replicate block
class Source1 hidden
class Source2 hidden
class Sink1 hidden
class Sink2 hidden
```

```bash
protoflow execute Replicate
```

#### [`Sort`]

Sorts a single input message stream in ascending order.

```mermaid
block-beta
columns 7
Source space:2 Sort space:2 Sink
Source-- "input" -->Sort
Sort-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Sort block
class Source hidden
class Sink hidden
class Sink2 hidden
```

```bash
protoflow execute Sort
```

#### [`Split`]

Divides a single input message stream into multiple output streams using a round-robin approach.

```mermaid
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Split space:3
space:5 Sink2 space:1
Source-- "input" -->Split
Split-- "output_1" -->Sink1
Split-- "output_2" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Split block
class Source hidden
class Sink1 hidden
class Sink2 hidden
```

```bash
protoflow execute Split
```

#### [`SplitString`]

A block that splits the received input message, with an optional delimiter string parameter
Expand Down Expand Up @@ -795,6 +929,7 @@ To add a new block type implementation, make sure to examine and amend:
[`examples`]: lib/protoflow/examples

[`Buffer`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Buffer.html
[`Concat`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Concat.html
[`ConcatStrings`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ConcatStrings.html
[`Const`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Const.html
[`Count`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Count.html
Expand All @@ -809,12 +944,16 @@ To add a new block type implementation, make sure to examine and amend:
[`EncodeHex`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.EncodeHex.html
[`EncodeJSON`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.EncodeJson.html
[`Hash`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Hash.html
[`Merge`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Merge.html
[`Random`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Random.html
[`ReadDir`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadDir.html
[`ReadEnv`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadEnv.html
[`ReadFile`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadFile.html
[`ReadSocket`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadSocket.html
[`ReadStdin`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.ReadStdin.html
[`Replicate`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Replicate.html
[`Sort`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Sort.html
[`Split`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.Split.html
[`SplitString`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.SplitString.html
[`WriteFile`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.WriteFile.html
[`WriteSocket`]: https://docs.rs/protoflow-blocks/latest/protoflow_blocks/struct.WriteSocket.html
Expand Down
15 changes: 15 additions & 0 deletions lib/protoflow-blocks/doc/flow/concat.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
block-beta
columns 7
space:1 Source1 space:5
space:3 Concat space:1 Sink space:1
space:1 Source2 space:5
Source1-- "input" -->Concat
Source2-- "input" -->Concat
Concat-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Concat block
class Source1 hidden
class Source2 hidden
class Sink hidden
28 changes: 28 additions & 0 deletions lib/protoflow-blocks/doc/flow/concat.seq.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
sequenceDiagram
autonumber
participant BlockA as Another block
participant BlockA2 as Another block
participant Concat.input as Concat.input port
participant Concat.input2 as Concat.input port
participant Concat as Concat block
participant Concat.output as Concat.output port
participant BlockB as Another block

BlockA-->>Concat: Connect
BlockA2-->>Concat: Connect
Concat-->>BlockB: Connect

loop Concat process
BlockA->>Concat: Message
Concat->>Concat: Store message
BlockA2->>Concat: Message
Concat->>Concat: Store message
end
Concat->>Concat: Concat messages
Concat->>BlockB: Message
BlockA-->>Concat: Disconnect
BlockA2-->>Concat: Disconnect
Concat-->>Concat.input: Close
Concat-->>Concat.input2: Close
Concat-->>Concat.output: Close
Concat-->>BlockB: Disconnect
15 changes: 15 additions & 0 deletions lib/protoflow-blocks/doc/flow/merge.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
block-beta
columns 7
space:1 Source1 space:5
space:3 Merge space:1 Sink space:1
space:1 Source2 space:5
Source1-- "input" -->Merge
Source2-- "input" -->Merge
Merge-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Merge block
class Source1 hidden
class Source2 hidden
class Sink hidden
26 changes: 26 additions & 0 deletions lib/protoflow-blocks/doc/flow/merge.seq.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
sequenceDiagram
autonumber
participant BlockA as Another block
participant BlockA2 as Another block
participant Merge.input as Merge.input port
participant Merge.input2 as Merge.input port
participant Merge as Merge block
participant Merge.output as Merge.output port
participant BlockB as Another block

BlockA-->>Merge: Connect
BlockA2-->>Merge: Connect
Merge-->>BlockB: Connect

loop Merge process
BlockA->>Merge: Message
Merge->>BlockB: Message
BlockA2->>Merge: Message
Merge->>BlockB: Message
end
BlockA-->>Merge: Disconnect
BlockA2-->>Merge: Disconnect
Merge-->>Merge.input: Close
Merge-->>Merge.input2: Close
Merge-->>Merge.output: Close
Merge-->>BlockB: Disconnect
17 changes: 17 additions & 0 deletions lib/protoflow-blocks/doc/flow/replicate.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Replicate space:3
space:5 Sink2 space:1

Source-- "input" -->Replicate
Replicate-- "output" -->Sink1
Replicate-- "output" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Replicate block
class Source1 hidden
class Source2 hidden
class Sink1 hidden
class Sink2 hidden
27 changes: 27 additions & 0 deletions lib/protoflow-blocks/doc/flow/replicate.seq.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
sequenceDiagram
autonumber
participant BlockA as Another block
participant Replicate.input as Replicate.input port
participant Replicate as Replicate block
participant Replicate.output_1 as Replicate.output port
participant BlockB as Another block
participant Replicate.output_2 as Replicate.output port
participant BlockC as Another block

BlockA-->>Replicate: Connect
Replicate-->>BlockB: Connect
Replicate-->>BlockC: Connect

loop Replicate process
BlockA->>Replicate: Message
Replicate->>BlockB: Message
Replicate->>BlockC: Message

end

BlockA-->>Replicate: Disconnect
Replicate-->>Replicate.input: Close
Replicate-->>Replicate.output_1: Close
Replicate-->>BlockB: Disconnect
Replicate-->>Replicate.output_2: Close
Replicate-->>BlockC: Disconnect
11 changes: 11 additions & 0 deletions lib/protoflow-blocks/doc/flow/sort.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
block-beta
columns 7
Source space:2 Sort space:2 Sink
Source-- "input" -->Sort
Sort-- "output" -->Sink

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Sort block
class Source hidden
class Sink hidden
22 changes: 22 additions & 0 deletions lib/protoflow-blocks/doc/flow/sort.seq.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
sequenceDiagram
autonumber
participant BlockA as Another block
participant Sort.input as Sort.input port
participant Sort as Sort block
participant Sort.output as Sort.output port
participant BlockB as Another block

BlockA-->>Sort: Connect
Sort-->>BlockB: Connect

loop Sort process
BlockA->>Sort: Message
Sort->>Sort: Store message
end

Sort->>Sort: Sort messages
Sort->>BlockB: Message
BlockA-->>Sort: Disconnect
Sort-->>Sort.input: Close
Sort-->>Sort.output: Close
Sort-->>BlockB: Disconnect
15 changes: 15 additions & 0 deletions lib/protoflow-blocks/doc/flow/split.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
block-beta
columns 7
space:5 Sink1 space:1
space:1 Source space:1 Split space:3
space:5 Sink2 space:1
Source-- "input" -->Split
Split-- "output_1" -->Sink1
Split-- "output_2" -->Sink2

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Split block
class Source hidden
class Sink1 hidden
class Sink2 hidden
Loading
Loading