Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example flows (top-level) / (metacrate) #24

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions examples/csv_roundtrip.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
let csv_content = s.const_string("Name,Age,Score\nAlice,25,90\nBob,30,85\nCharlie,22,95");
let line_encoder = s.encode_lines();
let decoder = s.decode_csv();
let encoder = s.encode_csv();
let output_path = s.const_string("output.csv");
let write_file = s.write_file();
s.connect(&csv_content.output, &line_encoder.input);
s.connect(&line_encoder.output, &decoder.input);
s.connect(&decoder.header, &encoder.header);
s.connect(&decoder.rows, &encoder.rows);
s.connect(&output_path.output, &write_file.path);
s.connect(&encoder.output, &write_file.input);
})
}
23 changes: 23 additions & 0 deletions examples/env_var_inspector.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
let name_param = s.const_string("PATH");
let env_reader = s.read_env();
let split_var = s.split_string(":");
let line_encoder = s.encode_lines();
let stdout = s.write_stdout();
s.connect(&name_param.output, &env_reader.name);
s.connect(&env_reader.output, &split_var.input);
s.connect(&split_var.output, &line_encoder.input);
s.connect(&line_encoder.output, &stdout.input);
})
}
23 changes: 23 additions & 0 deletions examples/hash_content.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
let content = s.const_string("Hello, World!");
let line_encoder = s.encode_lines();
let hash_content = s.hash_sha2();
Comment on lines +13 to +15
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not able to run this file directly but copying the code to somewhere else you get

$ echo "Hello, World!" | sha256sum
c98c24b677eff44860afea6f493bbaec5bb1c4cbb209c6fc2bbb47f66ff2ad31

instead of the correct

$ echo -n "Hello, World!" | sha256sum
dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f

due to the s.encode_lines() adding a newline that also gets hashed.

let encode_hex = s.encode_hex();
let stdout = s.write_stdout();
s.connect(&content.output, &line_encoder.input);
s.connect(&line_encoder.output, &hash_content.input);
s.connect(&hash_content.hash, &encode_hex.input);
s.connect(&encode_hex.output, &stdout.input);
})
}
31 changes: 31 additions & 0 deletions examples/json_sanitizer.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
let json_content = s.const_string(
r#"{
"Name": "Alice",
"Age": 25,
"Score": 90
}"#,
);
let line_encoder = s.encode_lines();
let decoder = s.decode_json();
let encoder = s.encode_json();
let sanitized_path = s.const_string("sanitized.json");
let write_file = s.write_file();
s.connect(&json_content.output, &line_encoder.input);
s.connect(&line_encoder.output, &decoder.input);
s.connect(&decoder.output, &encoder.input);
s.connect(&sanitized_path.output, &write_file.path);
s.connect(&encoder.output, &write_file.input);
})
}
26 changes: 26 additions & 0 deletions examples/log_directories.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
// The path to a directory
let dir_path = s.const_string("/");
let dir_reader = s.read_dir();
let concat_paths = s.concat_strings_by("\n");
let line_encoder = s.encode_lines();
let write_path = s.const_string("log_directories.txt");
let write_file = s.write_file();
s.connect(&dir_path.output, &dir_reader.path);
s.connect(&dir_reader.output, &concat_paths.input);
s.connect(&concat_paths.output, &line_encoder.input);
s.connect(&write_path.output, &write_file.path);
s.connect(&line_encoder.output, &write_file.input);
})
}
27 changes: 27 additions & 0 deletions examples/write_and_read_file.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env rust-script
//! This is free and unencumbered software released into the public domain.
//!
//! ```cargo
//! [dependencies]
//! protoflow = "0.4.3"
//! ```

use protoflow::{blocks::*, BlockResult};

fn main() -> BlockResult {
System::run(|s| {
let file_content = s.const_string("/home/user/documents\n/home/user/downloads\n/home/user/music\n/home/user/pictures\n/home/user/videos");
let line_encoder = s.encode_lines();
let content_path = s.const_string("content.txt");
let write_file = s.write_file();
s.connect(&file_content.output, &line_encoder.input);
s.connect(&content_path.output, &write_file.path);
s.connect(&line_encoder.output, &write_file.input);

let file_path = s.const_string("content.txt");
let reader = s.read_file();
let stdout = s.write_stdout();
s.connect(&file_path.output, &reader.path);
s.connect(&reader.output, &stdout.input);
})
}
29 changes: 29 additions & 0 deletions lib/protoflow/examples/csv_processing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# CSV Processing Example

This is a simple five-block example program that starts with a constant string
containing CSV data, encodes it into a byte stream, decodes it back into
structured CSV data consisting of headers and rows, re-encodes that structured
data back into CSV format, and writes the result to standard output (stdout).

Note that this program ensures the integrity of the CSV data during the decoding
and re-encoding process and demonstrates the handling of structured data in CSV format.

## Block Diagram

```mermaid
block-beta
columns 13
Const space:2 Encode space:2 DecodeCSV space:2 EncodeCSV space:2 WriteStdout
Const-- "output → input" -->Encode
Encode-- "output → input" -->DecodeCSV
DecodeCSV-- "header,rows, → header,rows" -->EncodeCSV
EncodeCSV-- "output → input" -->WriteStdout

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Const block
class Encode block
class DecodeCSV block
class EncodeCSV block
class WriteStdout block
```
22 changes: 22 additions & 0 deletions lib/protoflow/examples/csv_processing/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// This is free and unencumbered software released into the public domain.

use protoflow::{blocks::*, BlockResult};

pub fn main() -> BlockResult {
System::run(|s| {
let csv = s.const_string("Name,Age,Score\nAlice,25,90\nBob,30,85\nCharlie,22,95");

let encoder = s.encode_lines();
s.connect(&csv.output, &encoder.input);

let csv_decoder = s.decode_csv();
s.connect(&encoder.output, &csv_decoder.input);

let csv_encoder = s.encode_csv();
s.connect(&csv_decoder.header, &csv_encoder.header);
s.connect(&csv_decoder.rows, &csv_encoder.rows);

let stdout = s.write_stdout();
s.connect(&csv_encoder.output, &stdout.input);
})
}
27 changes: 27 additions & 0 deletions lib/protoflow/examples/decode_hex/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Hex Decode Example

This is a simple four-block example program that starts with a constant string
containing hexadecimal-encoded data, encodes it into a byte stream,
decodes the hexadecimal back into raw bytes, and writes the decoded data to
standard output (stdout).

Note that this program demonstrates how to handle hexadecimal input without
relying on external files or sources, making it fully self-contained.

## Block Diagram

```mermaid
block-beta
columns 10
Const space:2 Encode space:2 DecodeHex space:2 WriteStdout
Const-- "output → input" -->Encode
Encode-- "output → input" -->DecodeHex
DecodeHex-- "output → input" -->WriteStdout

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Const block
class Encode block
class DecodeHex block
class WriteStdout block
```
18 changes: 18 additions & 0 deletions lib/protoflow/examples/decode_hex/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// This is free and unencumbered software released into the public domain.

use protoflow::{blocks::*, BlockResult};

pub fn main() -> BlockResult {
System::run(|s| {
let input_string = s.const_string("54686520717569636b2062726f776e20666f78206a756d7073206f76657220746865206c617a7920646f670a");

let encoder = s.encode_lines();
s.connect(&input_string.output, &encoder.input);

let hex_decoder = s.decode_hex();
s.connect(&encoder.output, &hex_decoder.input);
Comment on lines +7 to +13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same problem, with interesting behaviour with DecodeHex! This appends a newline at the end before sending it to DecodeHex which should result in an error. But DecodeHex uses Bytes::chunks_exact(2) which leaves the newline character alone as it's one byte:

The chunks are slices and do not overlap. If chunk_size does not divide the length of the
slice, then the last up to chunk_size-1 elements will be omitted and can be retrieved
from the remainder function of the iterator.

I think that DecodeHex shouldn't accept and ignore the last byte silently so we should make a PR so that it checks the content length is divisible by two.


let stdout = s.write_stdout();
s.connect(&hex_decoder.output, &stdout.input);
})
}
29 changes: 29 additions & 0 deletions lib/protoflow/examples/delay_output/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Delayed Line Output Example

This is a simple five-block example program that starts with a constant string
containing multiple lines, splits the string into individual lines,
introduces a delay between processing each line, and writes the delayed lines
to standard output (stdout).

Note that this program demonstrates how to control the timing of data processing,
making it suitable for simulating real-time output or throttling data streams

## Block Diagram

```mermaid
block-beta
columns 13
Const space:2 SplitString space:2 Delay space:2 Encode space:2 WriteStdout
Const-- "output → input" -->SplitString
SplitString-- "output → input" -->Delay
Delay-- "output → input" -->Encode
Encode-- "output → input" -->WriteStdout

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Const block
class SplitString block
class Delay block
class Encode block
class WriteStdout block
```
21 changes: 21 additions & 0 deletions lib/protoflow/examples/delay_output/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// This is free and unencumbered software released into the public domain.

use protoflow::{blocks::*, BlockResult};

pub fn main() -> BlockResult {
System::run(|s| {
let input_string = s.const_string("Line 1\nLine 2\nLine 3\nLine 4");

let split_string = s.split_string("\n");
s.connect(&input_string.output, &split_string.input);

let delay = s.delay();
s.connect(&split_string.output, &delay.input);

let encoder = s.encode_lines();
s.connect(&delay.output, &encoder.input);

let stdout = s.write_stdout();
s.connect(&encoder.output, &stdout.input);
})
}
27 changes: 27 additions & 0 deletions lib/protoflow/examples/encode_hex/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Hex Encode Example

This is a simple four-block example program that starts with a constant string,
encodes it into a byte stream, converts the bytes into hexadecimal format,
and writes the hexadecimal-encoded output to standard output (stdout).

Note that this program demonstrates how to transform human-readable text into
its hexadecimal representation, which can be useful for encoding data for secure
transmission or storage.

## Block Diagram

```mermaid
block-beta
columns 10
Const space:2 Encode space:2 EncodeHex space:2 WriteStdout
Const-- "output → input" -->Encode
Encode-- "output → input" -->EncodeHex
EncodeHex-- "output → input" -->WriteStdout

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Const block
class Encode block
class EncodeHex block
class WriteStdout block
```
18 changes: 18 additions & 0 deletions lib/protoflow/examples/encode_hex/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// This is free and unencumbered software released into the public domain.

use protoflow::{blocks::*, BlockResult};

pub fn main() -> BlockResult {
System::run(|s| {
let input_string = s.const_string("The quick brown fox jumps over the lazy dog");

let encoder = s.encode_lines();
s.connect(&input_string.output, &encoder.input);
Comment on lines +7 to +10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise for here as hex("$string\n") != hex("$string")


let hex_encoder = s.encode_hex();
s.connect(&encoder.output, &hex_encoder.input);

let stdout = s.write_stdout();
s.connect(&hex_encoder.output, &stdout.input);
})
}
30 changes: 30 additions & 0 deletions lib/protoflow/examples/json_processing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# JSON Processing Example

This is a simple five-block example program that starts with a constant string
containing JSON data, encodes it into a byte stream, decodes it back into
structured JSON data, re-encodes that structured data into JSON format,
and writes the re-encoded JSON data to standard output (stdout).

Note that this program demonstrates the complete processing cycle of JSON data,
ensuring that the data is parsed and re-encoded while preserving its original
structure.

## Block Diagram

```mermaid
block-beta
columns 13
Const space:2 Encode space:2 DecodeJSON space:2 EncodeJSON space:2 WriteStdout
Const-- "output → input" -->Encode
Encode-- "output → input" -->DecodeJSON
DecodeJSON-- "output, → input" -->EncodeJSON
EncodeJSON-- "output → input" -->WriteStdout

classDef block height:48px,padding:8px;
classDef hidden visibility:none;
class Const block
class Encode block
class DecodeJSON block
class EncodeJSON block
class WriteStdout block
```
Loading