diff --git a/.cargo/config.toml b/.cargo/config.toml new file mode 100644 index 00000000..35049cbc --- /dev/null +++ b/.cargo/config.toml @@ -0,0 +1,2 @@ +[alias] +xtask = "run --package xtask --" diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md new file mode 100644 index 00000000..dae28f50 --- /dev/null +++ b/.github/ISSUE_TEMPLATE.md @@ -0,0 +1,3 @@ +## 👉 [Please follow one of these issue templates](https://github.com/monadicus/snarkops/issues/new/choose) 👈 + +**NOTE**: In order to keep the backlog clean and actionable, issues may be immediately closed if they do not follow one of the above templates. diff --git a/.github/ISSUE_TEMPLATE/bug.md b/.github/ISSUE_TEMPLATE/bug.md new file mode 100644 index 00000000..570db8b1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.md @@ -0,0 +1,46 @@ +--- +name: 🐛 Bug Report +about: Submit a bug report if something isn't working +title: "🐛 " +labels: "type: bug" +--- + +## 🐛 Bug Report + + + +(Write your description here) + +## Steps to Reproduce + + + +#### Code snippet to reproduce + +``` +# Add code here if relevent +``` + +#### Stack trace & error message + +``` +// Paste the output here +``` + +## Expected Behavior + + + +## Your Environment + +- +- +- diff --git a/.github/ISSUE_TEMPLATE/chore.md b/.github/ISSUE_TEMPLATE/chore.md new file mode 100644 index 00000000..deab7e6b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/chore.md @@ -0,0 +1,25 @@ +--- +name: 🔧 Chore +about: Submit a behind-the-scenes change request +title: "🔧 " +labels: "type: chore" +--- + +## 🔧 Chore + + + +(Write what chore needs to be done) + +## Motivation + + + +(Outline your motivation here) + +**Are you willing to open a pull request?** (See +[CONTRIBUTING](../../CONTRIBUTING.md)) diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..ec4bb386 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/documentation.md b/.github/ISSUE_TEMPLATE/documentation.md new file mode 100644 index 00000000..78611203 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -0,0 +1,14 @@ +--- +name: 📚 Documentation +about: Report an issue related to documentation +title: "📚 " +labels: "type: documentation" +--- + +## 📚 Documentation + + diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.md new file mode 100644 index 00000000..106f7581 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.md @@ -0,0 +1,33 @@ +--- +name: ✨ Feature +about: Submit a new feature request +title: "✨ " +labels: "type: feature" +--- + +## ✨ Feature + + + +## Motivation + + + +## Implementation + + + +**Are you willing to open a pull request?** (See +[CONTRIBUTING](../../CONTRIBUTING.md)) diff --git a/.github/ISSUE_TEMPLATE/refactor.md b/.github/ISSUE_TEMPLATE/refactor.md new file mode 100644 index 00000000..3041e4dc --- /dev/null +++ b/.github/ISSUE_TEMPLATE/refactor.md @@ -0,0 +1,31 @@ +--- +name: ♻️ Refactor +about: Submit a code cleanup request +title: "♻️ " +labels: "type: refactor" +--- + +## ♻️ Refactor + + + +## Motivation + + + +## Implementation + + + +**Are you willing to open a pull request?** (See +[CONTRIBUTING](../../CONTRIBUTING.md)) diff --git a/.github/ISSUE_TEMPLATE/task.md b/.github/ISSUE_TEMPLATE/task.md new file mode 100644 index 00000000..15d7eeb7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/task.md @@ -0,0 +1,30 @@ +--- +name: ✅ Task +about: Submit a new Task +title: "✅ " +labels: "type: task" +--- + +## ✅ Task + + + +## Motivation + + + +## Details + + + +**Are you willing to open a pull request?** (See +[CONTRIBUTING](../../CONTRIBUTING.md)) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..4dffb7b6 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,34 @@ + + +## Motivation + +(Write your motivation here) + +## Explanation of Changes + + + +(Write your explanation here) + +## Testing + + + +(Write your test plan here) + +## Related PRs and Issues + + + +(Link your related PRs and Issues here) diff --git a/.github/workflows/mdbook.yml b/.github/workflows/mdbook.yml new file mode 100644 index 00000000..ee79dfc6 --- /dev/null +++ b/.github/workflows/mdbook.yml @@ -0,0 +1,40 @@ +name: MDBook Deploy +on: + push: + branches: + - main + +permissions: + contents: write # To push a branch + pages: write # To push to a GitHub Pages site + id-token: write # To update the deployment status + +concurrency: + group: "pages" + cancel-in-progress: false + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Install latest mdbook + run: | + tag=$(curl 'https://api.github.com/repos/rust-lang/mdbook/releases/latest' | jq -r '.tag_name') + url="https://github.com/rust-lang/mdbook/releases/download/${tag}/mdbook-${tag}-x86_64-unknown-linux-gnu.tar.gz" + mkdir mdbook + curl -sSL $url | tar -xz --directory=./mdbook + echo `pwd`/mdbook >> $GITHUB_PATH + - name: Build Book + run: mdbook build + - name: Setup Pages + uses: actions/configure-pages@v5 + - name: Upload artifact + uses: actions/upload-pages-artifact@v3 + with: + path: "book" + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index 6e34b110..fc9afa03 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,16 @@ +# rust files **/target/ + +# snops files /tests -/snot-control-data -/snot-data /snops-control-data /snops-data /metrics-data /specs/local-*.yaml + +# vscode files .vscode/settings.json + +# mdbook files +book +index.html \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..538797c9 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,54 @@ +# Contributing + +This file describes the process for contributing to `snarkops`. + +## Starting + + + + +## Commits + +Your commits must follow specific guidelines. + + + + +### Convention + +All commits are to follow the +[Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) standard. +Commit messages should always be meaningful. + +## Getting Ready For a PR + +This section describes actions to keep in mind while developing. + +### Formatting and Cleanliness + +Please ensure your code is formatted and the formatting tool gives no warnings(if using a local snarkos/vm you can ignore warnings given from those repos). + +## PRs + +For creating the PR, please follow the instructions below. + +1. Firstly, please open a + [PR](https://github.com/monadicus/snarkops/pulls) from your branch + to the `main` branch of `snarkops`. +2. Please fill in the PR template that is there. +3. Then assign it to yourself and anyone else who worked on the issue with you. +4. Make sure all CI tests pass. +5. Finally, please assign at least two of the following reviewers to your PR: + - [gluax](https://github.com/gluax) + - [Meshiest](https://github.com/Meshiest) + - [voximity](https://github.com/voximity) diff --git a/Cargo.lock b/Cargo.lock index fa09cd5c..105a49bc 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -547,6 +547,15 @@ dependencies = [ "clap_derive", ] +[[package]] +name = "clap-markdown" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "325f50228f76921784b6d9f2d62de6778d834483248eefecd27279174797e579" +dependencies = [ + "clap", +] + [[package]] name = "clap_builder" version = "4.5.2" @@ -586,6 +595,16 @@ version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "98cc8fbded0c607b7ba9dd60cd98df59af97e84d24e49c8557331cfc26d301ce" +[[package]] +name = "clap_mangen" +version = "0.2.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1dd95b5ebb5c1c54581dd6346f3ed6a79a3eef95dd372fc2ac13d535535300e" +dependencies = [ + "clap", + "roff", +] + [[package]] name = "colorchoice" version = "1.0.0" @@ -2956,6 +2975,12 @@ dependencies = [ "librocksdb-sys", ] +[[package]] +name = "roff" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b833d8d034ea094b1ea68aa6d5c740e0d04bad9d16568d08ba6f76823a114316" + [[package]] name = "rust_decimal" version = "1.35.0" @@ -4657,7 +4682,9 @@ version = "0.1.0" dependencies = [ "anyhow", "clap", + "clap-markdown", "clap_complete", + "clap_mangen", "reqwest 0.12.3", "serde_json", "snops-common", @@ -4667,8 +4694,11 @@ dependencies = [ name = "snops-common" version = "0.1.0" dependencies = [ + "anyhow", "checkpoint", "clap", + "clap-markdown", + "clap_mangen", "futures", "http 1.1.0", "lasso", @@ -5816,6 +5846,29 @@ dependencies = [ "windows-sys 0.48.0", ] +[[package]] +name = "xshell" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6db0ab86eae739efd1b054a8d3d16041914030ac4e01cd1dca0cf252fd8b6437" +dependencies = [ + "xshell-macros", +] + +[[package]] +name = "xshell-macros" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d422e8e38ec76e2f06ee439ccc765e9c6a9638b9e7c9f2e8255e4d41e8bd852" + +[[package]] +name = "xtask" +version = "0.1.0" +dependencies = [ + "anyhow", + "xshell", +] + [[package]] name = "xtasks" version = "0.0.2" diff --git a/Cargo.toml b/Cargo.toml index 08108ead..ee53c457 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -6,6 +6,7 @@ members = [ "crates/snops-cli", "crates/snops-common", "crates/checkpoint", + "crates/xtask", ] resolver = "2" @@ -51,6 +52,8 @@ checkpoint = { path = "./crates/checkpoint" } chrono = { version = "0.4", features = ["now"], default-features = false } clap = { version = "4.5", features = ["derive"] } clap_complete = { version = "4.5" } +clap_mangen = { version = "0.2" } +clap-markdown = "0.1" colored = "2" crossterm = { version = "0.27", default-features = false } dashmap = "5.5" diff --git a/README.md b/README.md index de2301ca..5a45117a 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,36 @@ -# snarkos-test + -Crates for AOT transaction generation and repeatable testing infrastructure. +

+ SNOPS +

-Requires a local clone of `snarkos` and `snarkvm` in the parent directory. That -is, your file tree should look like: +This repository is home to the `snops` (snarkOS operations) ecosystem, and +`snarkos-aot`, a crate for performing ahead-of-time ledger actions. -``` -- - - snarkos - - snarkos-test - - snarkvm -``` +snops is a suite of tools that can be used to maintain [Aleo](https://aleo.org/) +network environments. The environments are defined in a manner similar to +infrastructure-as-code, which means you can create repeatable infrastructure +using the environment schema. -## Scripts +This can be used to create devnets, simulate events on the network like outages +and attacks, and guarantee metrics. -In order to use the [test scripts](/scripts/), you must build `snarkos` in -release mode with `cargo build --release` from the `snarkos` directory. +To learn more about `snops` we recommend checking out the mdbook [here](todo). + +## Contributing + +`snops` is free and open source. You can find the source code on +[GitHub](https://github.com/monadicus/snarkops) and issues and feature requests can be posted on +the [GitHub issue tracker](https://github.com/monadicus/snarkops/issues). If you'd like to contribute, please read +the [CONTRIBUTING](https://github.com/monadicus/snarkops/blob/main/CONTRIBUTING.md) guide and consider opening +a [pull request](https://github.com/monadicus/snarkops/pulls). + +## License + +The `snops` source and documentation are released under +the [MIT License +](https://github.com/monadicus/snarkops/blob/main/LICENSE). diff --git a/book.toml b/book.toml new file mode 100644 index 00000000..eed15b0b --- /dev/null +++ b/book.toml @@ -0,0 +1,6 @@ +[book] +authors = ["monadicus"] +language = "en" +multilingual = false +src = "snops_book" +title = "snops" diff --git a/crates/aot/Cargo.toml b/crates/aot/Cargo.toml index 8ff84438..cd99699c 100644 --- a/crates/aot/Cargo.toml +++ b/crates/aot/Cargo.toml @@ -17,6 +17,9 @@ node = [ "num_cpus", "rayon", ] +docpages = ["clipages", "mangen"] +clipages = ["snops-common/clipages"] +mangen = ["snops-common/mangen"] [dependencies] aleo-std.workspace = true diff --git a/crates/aot/src/cli.rs b/crates/aot/src/cli.rs index 66940321..9ad34d1b 100644 --- a/crates/aot/src/cli.rs +++ b/crates/aot/src/cli.rs @@ -5,6 +5,8 @@ use std::io::BufWriter; use std::{io, path::PathBuf, thread}; use anyhow::Result; +#[cfg(any(feature = "clipages", feature = "mangen"))] +use clap::CommandFactory; use clap::Parser; use crossterm::tty::IsTty; use reqwest::Url; @@ -19,7 +21,7 @@ use crate::{ }; #[derive(Debug, Parser)] -#[clap(name = "snarkOS AoT", author = "MONADIC.US")] +#[clap(author = "MONADIC.US")] pub struct Cli { #[arg(long)] pub enable_profiling: bool, @@ -46,6 +48,10 @@ pub enum Command { Execute(Execute), #[command(subcommand)] Authorize(Authorize), + #[cfg(feature = "mangen")] + Man(snops_common::mangen::Mangen), + #[cfg(feature = "clipages")] + Md(snops_common::clipages::Clipages), } pub trait Flushable { @@ -246,6 +252,14 @@ impl Cli { println!("{}", serde_json::to_string(&command.parse()?)?); Ok(()) } + #[cfg(feature = "mangen")] + Command::Man(mangen) => mangen.run( + Cli::command(), + env!("CARGO_PKG_VERSION"), + env!("CARGO_PKG_NAME"), + ), + #[cfg(feature = "clipages")] + Command::Md(clipages) => clipages.run::(env!("CARGO_PKG_NAME")), } } } diff --git a/crates/snops-agent/Cargo.toml b/crates/snops-agent/Cargo.toml index 0757f428..6e8ac510 100644 --- a/crates/snops-agent/Cargo.toml +++ b/crates/snops-agent/Cargo.toml @@ -3,6 +3,12 @@ name = "snops-agent" version = "0.1.0" edition = "2021" +[features] +default = [] +docpages = ["clipages", "mangen"] +clipages = ["snops-common/clipages"] +mangen = ["snops-common/mangen"] + [dependencies] anyhow.workspace = true bincode.workspace = true diff --git a/crates/snops-agent/src/cli.rs b/crates/snops-agent/src/cli.rs index b0c89a08..77f2f158 100644 --- a/crates/snops-agent/src/cli.rs +++ b/crates/snops-agent/src/cli.rs @@ -4,6 +4,8 @@ use std::{ path::PathBuf, }; +#[cfg(any(feature = "clipages", feature = "mangen"))] +use clap::CommandFactory; use clap::Parser; use http::Uri; use snops_common::state::{AgentId, AgentMode, PortConfig}; @@ -54,9 +56,44 @@ pub struct Cli { #[clap(short, long, default_value_t = false)] /// Run the agent in quiet mode, suppressing most node output pub quiet: bool, + + #[cfg(any(feature = "clipages", feature = "mangen"))] + #[clap(subcommand)] + pub command: Commands, +} + +#[cfg(any(feature = "clipages", feature = "mangen"))] +#[derive(Debug, Parser)] +pub enum Commands { + #[cfg(feature = "mangen")] + Man(snops_common::mangen::Mangen), + #[cfg(feature = "clipages")] + Md(snops_common::clipages::Clipages), } impl Cli { + #[cfg(any(feature = "clipages", feature = "mangen"))] + pub fn run(self) { + match self.command { + #[cfg(feature = "mangen")] + Commands::Man(mangen) => { + mangen + .run( + Cli::command(), + env!("CARGO_PKG_VERSION"), + env!("CARGO_PKG_NAME"), + ) + .unwrap(); + } + #[cfg(feature = "clipages")] + Commands::Md(clipages) => { + clipages.run::(env!("CARGO_PKG_NAME")).unwrap(); + } + } + + std::process::exit(0); + } + pub fn endpoint_and_uri(&self) -> (String, Uri) { // get the endpoint let endpoint = self diff --git a/crates/snops-agent/src/main.rs b/crates/snops-agent/src/main.rs index 9785b440..ad2bb197 100644 --- a/crates/snops-agent/src/main.rs +++ b/crates/snops-agent/src/main.rs @@ -70,6 +70,9 @@ async fn main() { .try_init() .unwrap(); + // For documentation purposes will exit after running the command. + #[cfg(any(feature = "clipages", feature = "mangen"))] + Cli::parse().run(); let args = Cli::parse(); // get the network interfaces available to this node diff --git a/crates/snops-cli/Cargo.toml b/crates/snops-cli/Cargo.toml index dbb9bf5e..83cef208 100644 --- a/crates/snops-cli/Cargo.toml +++ b/crates/snops-cli/Cargo.toml @@ -3,7 +3,11 @@ name = "snops-cli" version = "0.1.0" edition = "2021" -# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html +[features] +default = [] +docpages = ["clipages", "mangen"] +clipages = ["snops-common/clipages"] +mangen = ["snops-common/mangen"] [dependencies] anyhow.workspace = true @@ -12,3 +16,13 @@ clap_complete.workspace = true reqwest = { workspace = true, features = ["blocking", "json"] } serde_json.workspace = true snops-common.workspace = true + +[build-dependencies] +anyhow.workspace = true +clap.workspace = true +clap_complete.workspace = true +clap_mangen = { workspace = true, optional = true } +clap-markdown.workspace = true +reqwest = { workspace = true, features = ["blocking", "json"] } +serde_json.workspace = true +snops-common.workspace = true diff --git a/crates/snops-cli/src/cli.rs b/crates/snops-cli/src/cli.rs index 978a25a0..7378db00 100644 --- a/crates/snops-cli/src/cli.rs +++ b/crates/snops-cli/src/cli.rs @@ -9,7 +9,7 @@ pub struct Cli { url: String, /// The subcommand to run. #[clap(subcommand)] - pub subcommand: crate::commands::Commands, + pub subcommand: crate::Commands, } impl Cli { diff --git a/crates/snops-cli/src/commands/agent.rs b/crates/snops-cli/src/commands/agent.rs index 54310a8e..fa0b7b0f 100644 --- a/crates/snops-cli/src/commands/agent.rs +++ b/crates/snops-cli/src/commands/agent.rs @@ -6,21 +6,21 @@ use reqwest::blocking::{Client, Response}; use snops_common::state::AgentId; use super::DUMMY_ID; -use crate::cli::Cli; +use crate::Cli; -/// For interacting with snop environments. +/// For interacting with snop agents. #[derive(Debug, Parser)] pub struct Agent { /// Show a specific agent's info. #[clap(value_hint = ValueHint::Other, default_value = DUMMY_ID)] id: AgentId, #[clap(subcommand)] - command: Commands, + command: AgentCommands, } /// Env commands #[derive(Debug, Parser)] -enum Commands { +enum AgentCommands { /// Get the specific agent. #[clap(alias = "i")] Info, @@ -36,7 +36,7 @@ enum Commands { impl Agent { pub fn run(self, url: &str, client: Client) -> Result { - use Commands::*; + use AgentCommands::*; Ok(match self.command { List => { let ep = format!("{url}/api/v1/agents"); diff --git a/crates/snops-cli/src/commands/env/mod.rs b/crates/snops-cli/src/commands/env/mod.rs index 6f93b38c..e630e07f 100644 --- a/crates/snops-cli/src/commands/env/mod.rs +++ b/crates/snops-cli/src/commands/env/mod.rs @@ -14,12 +14,12 @@ pub struct Env { #[clap(default_value = "default", value_hint = ValueHint::Other)] id: String, #[clap(subcommand)] - command: Commands, + command: EnvCommands, } /// Env commands #[derive(Debug, Parser)] -enum Commands { +enum EnvCommands { /// Get an env's specific agent by. #[clap(alias = "a")] Agent { @@ -65,25 +65,11 @@ enum Commands { /// Get an env's storage info. #[clap(alias = "store")] Storage, - - /// Start an environment's timeline (a test). - Start { - /// Start a specific timeline. - #[clap(value_hint = ValueHint::Other)] - timeline_id: String, - }, - - /// Stop an environment's timeline. - Stop { - /// Stop a specific timeline. - #[clap(value_hint = ValueHint::Other)] - timeline_id: String, - }, } impl Env { pub fn run(self, url: &str, client: Client) -> Result { - use Commands::*; + use EnvCommands::*; Ok(match self.command { Agent { key } => { let ep = format!("{url}/api/v1/env/{}/agents/{}", self.id, key); @@ -127,16 +113,6 @@ impl Env { client.get(ep).send()? } - Start { timeline_id } => { - let ep = format!("{url}/api/v1/env/{}/timelines/{timeline_id}", self.id); - - client.post(ep).send()? - } - Stop { timeline_id } => { - let ep = format!("{url}/api/v1/env/{}/timelines/{timeline_id}", self.id); - - client.delete(ep).send()? - } }) } } diff --git a/crates/snops-cli/src/commands/env/timeline.rs b/crates/snops-cli/src/commands/env/timeline.rs index 4d7f7108..7f13f084 100644 --- a/crates/snops-cli/src/commands/env/timeline.rs +++ b/crates/snops-cli/src/commands/env/timeline.rs @@ -5,26 +5,26 @@ use clap::{error::ErrorKind, CommandFactory, Parser, ValueHint}; use reqwest::blocking::{Client, Response}; use snops_common::state::TimelineId; -use crate::{cli::Cli, commands::DUMMY_ID}; +use crate::{Cli, DUMMY_ID}; -/// For interacting with snop environments. +/// For interacting with snop environment timelines. #[derive(Debug, Parser)] pub struct Timeline { /// The timeline id. #[clap(value_hint = ValueHint::Other, default_value = DUMMY_ID)] id: TimelineId, #[clap(subcommand)] - command: Commands, + command: TimelineCommands, } -/// Env commands +/// Timeline commands #[derive(Debug, Parser)] -enum Commands { +enum TimelineCommands { /// Apply a timeline to an environment. #[clap(alias = "a")] Apply, - /// Delete a timeline from an environment.zs + /// Delete a timeline from an environment. #[clap(alias = "d")] Delete, @@ -36,11 +36,17 @@ enum Commands { /// Timeline id is ignored. #[clap(alias = "ls")] List, + + /// Start an environment's timeline (a test). + Start, + + /// Stop an environment's timeline. + Stop, } impl Timeline { pub fn run(self, url: &str, env_id: &str, client: Client) -> Result { - use Commands::*; + use TimelineCommands::*; Ok(match self.command { List => { let ep = format!("{url}/api/v1/env/{env_id}/timelines"); @@ -70,6 +76,16 @@ impl Timeline { client.get(ep).send()? } + Start => { + let ep = format!("{url}/api/v1/env/{env_id}/timelines/{}", self.id); + + client.post(ep).send()? + } + Stop => { + let ep = format!("{url}/api/v1/env/{env_id}/timelines/{}", self.id); + + client.delete(ep).send()? + } }) } } diff --git a/crates/snops-cli/src/commands/mod.rs b/crates/snops-cli/src/commands/mod.rs index ce2e917c..a89aa0cc 100644 --- a/crates/snops-cli/src/commands/mod.rs +++ b/crates/snops-cli/src/commands/mod.rs @@ -2,10 +2,10 @@ use anyhow::Result; use clap::{CommandFactory, Parser}; use serde_json::Value; -use crate::cli::Cli; +use crate::Cli; /// The dummy value for the ids to hack around the missing required argument. -static DUMMY_ID: &str = "dummy_value___"; +pub(crate) static DUMMY_ID: &str = "dummy_value___"; mod agent; mod env; @@ -22,6 +22,10 @@ pub enum Commands { Agent(agent::Agent), #[clap(alias = "e")] Env(env::Env), + #[cfg(feature = "mangen")] + Man(snops_common::mangen::Mangen), + #[cfg(feature = "clipages")] + Md(snops_common::clipages::Clipages), } impl Commands { @@ -39,6 +43,20 @@ impl Commands { Commands::Agent(agent) => agent.run(url, client), Commands::Env(env) => env.run(url, client), + #[cfg(feature = "mangen")] + Commands::Man(mangen) => { + mangen.run( + Cli::command(), + env!("CARGO_PKG_VERSION"), + env!("CARGO_PKG_NAME"), + )?; + return Ok(()); + } + #[cfg(feature = "clipages")] + Commands::Md(clipages) => { + clipages.run::(env!("CARGO_PKG_NAME"))?; + return Ok(()); + } }?; if !response.status().is_success() { diff --git a/crates/snops-cli/src/main.rs b/crates/snops-cli/src/main.rs index a02f40e9..e245398a 100644 --- a/crates/snops-cli/src/main.rs +++ b/crates/snops-cli/src/main.rs @@ -4,7 +4,10 @@ use anyhow::Result; use clap::Parser; mod cli; +pub(crate) use cli::*; + mod commands; +pub(crate) use commands::*; fn main() -> Result<()> { let cli = cli::Cli::parse(); diff --git a/crates/snops-common/Cargo.toml b/crates/snops-common/Cargo.toml index 734ed479..d061b996 100644 --- a/crates/snops-common/Cargo.toml +++ b/crates/snops-common/Cargo.toml @@ -3,9 +3,17 @@ name = "snops-common" version = "0.1.0" edition = "2021" +[features] +default = [] +clipages = ["anyhow", "clap-markdown"] +mangen = ["anyhow", "clap_mangen"] + [dependencies] +anyhow = { workspace = true, optional = true } checkpoint = { workspace = true, features = ["serde"] } clap.workspace = true +clap_mangen = { workspace = true, optional = true } +clap-markdown = { workspace = true, optional = true } futures.workspace = true http.workspace = true lasso.workspace = true diff --git a/crates/snops-common/src/clipages.rs b/crates/snops-common/src/clipages.rs new file mode 100644 index 00000000..62a04c00 --- /dev/null +++ b/crates/snops-common/src/clipages.rs @@ -0,0 +1,33 @@ +use std::{fs::OpenOptions, io::Write, path::PathBuf}; + +use anyhow::{Context, Result}; +use clap::{CommandFactory, Parser, ValueHint}; + +/// For generating cli markdown. +/// Only with the clipages feature enabled. +#[derive(Debug, Parser)] +pub struct Clipages { + #[clap(value_hint = ValueHint::Other, default_value = "snops_book/user_guide/clis")] + directory: PathBuf, +} + +impl Clipages { + pub fn run(self, pkg_name: &'static str) -> Result<()> { + std::fs::create_dir_all(&self.directory) + .with_context(|| format!("creating {:?}", self.directory))?; + + let pkg_name = pkg_name.to_string().to_uppercase().replace('-', "_"); + let file_path = self.directory.join(format!("{}.md", pkg_name)); + let mut md_file = OpenOptions::new() + .create(true) + .write(true) + .truncate(true) + .open(file_path) + .with_context(|| format!("opening {:?}", self.directory)) + .map(std::io::BufWriter::new)?; + let md_content = clap_markdown::help_markdown::(); + + md_file.write_all(md_content.as_bytes())?; + Ok(()) + } +} diff --git a/crates/snops-common/src/lib.rs b/crates/snops-common/src/lib.rs index 1523b757..52737511 100644 --- a/crates/snops-common/src/lib.rs +++ b/crates/snops-common/src/lib.rs @@ -5,6 +5,11 @@ pub use lasso; pub mod api; pub mod constant; +#[cfg(feature = "clipages")] +pub mod clipages; +#[cfg(feature = "mangen")] +pub mod mangen; + pub mod prelude { pub use crate::rpc::*; pub use crate::set::*; diff --git a/crates/snops-common/src/mangen.rs b/crates/snops-common/src/mangen.rs new file mode 100644 index 00000000..343d0707 --- /dev/null +++ b/crates/snops-common/src/mangen.rs @@ -0,0 +1,62 @@ +use std::{fs::OpenOptions, io::Write, path::PathBuf}; + +use anyhow::{Context, Result}; +use clap::{Command, Parser, ValueHint}; + +/// For generating cli manpages. +/// Only with the mangen feature enabled. +#[derive(Debug, Parser)] +pub struct Mangen { + #[clap(value_hint = ValueHint::Other, default_value = "target/man/snops-cli")] + directory: PathBuf, +} + +impl Mangen { + pub fn run(self, cmd: Command, version: &'static str, pkg_name: &'static str) -> Result<()> { + print_manpages(&self.directory, cmd, version, pkg_name)?; + Ok(()) + } +} + +fn print_manpages( + dir: &PathBuf, + cmd: Command, + version: &'static str, + pkg_name: &'static str, +) -> Result<()> { + let name = cmd.get_name(); + std::fs::create_dir_all(dir).with_context(|| format!("creating {dir:?}"))?; + let path = dir.join(format!("{name}.1")); + + let mut out = OpenOptions::new() + .create(true) + .write(true) + .truncate(true) + .open(&path) + .with_context(|| format!("opening {path:?}")) + .map(std::io::BufWriter::new)?; + clap_mangen::Man::new(cmd.clone()) + .title(pkg_name) + .section("1") + .source(format!("{pkg_name} {version}")) + .render(&mut out) + .with_context(|| format!("rendering {name}.1"))?; + out.flush().context("flushing man page")?; + drop(out); + + for subcmd in cmd.get_subcommands().filter(|c| !c.is_hide_set()) { + let subname = format!("{}-{}", name, subcmd.get_name()); + // SAFETY: Latest clap 4 requires names are &'static - this is + // not long-running production code, so we just leak the names here. + let subname = &*std::boxed::Box::leak(subname.into_boxed_str()); + let subcmd = subcmd.clone().name(subname).alias(subname).version(version); + print_manpages( + dir, + subcmd.clone().name(subname).version(version), + version, + pkg_name, + )?; + } + + Ok(()) +} diff --git a/crates/snops/Cargo.toml b/crates/snops/Cargo.toml index 1513821b..2fe6e885 100644 --- a/crates/snops/Cargo.toml +++ b/crates/snops/Cargo.toml @@ -3,6 +3,12 @@ name = "snops" version = "0.1.0" edition = "2021" +[features] +default = [] +docpages = ["clipages", "mangen"] +clipages = ["snops-common/clipages"] +mangen = ["snops-common/mangen"] + [dependencies] axum = { workspace = true, features = [ "http2", diff --git a/crates/snops/src/cli.rs b/crates/snops/src/cli.rs index cf720c16..1cade881 100644 --- a/crates/snops/src/cli.rs +++ b/crates/snops/src/cli.rs @@ -1,5 +1,7 @@ use std::{fmt::Display, path::PathBuf, str::FromStr}; +#[cfg(any(feature = "clipages", feature = "mangen"))] +use clap::CommandFactory; use clap::Parser; use url::Url; @@ -33,6 +35,43 @@ pub struct Cli { /// /// must contain http:// or https:// pub hostname: Option, + + #[cfg(any(feature = "clipages", feature = "mangen"))] + #[clap(subcommand)] + pub command: Commands, +} + +#[cfg(any(feature = "clipages", feature = "mangen"))] +#[derive(Debug, Parser)] +pub enum Commands { + #[cfg(feature = "mangen")] + Man(snops_common::mangen::Mangen), + #[cfg(feature = "clipages")] + Md(snops_common::clipages::Clipages), +} + +impl Cli { + #[cfg(any(feature = "clipages", feature = "mangen"))] + pub fn run(self) { + match self.command { + #[cfg(feature = "mangen")] + Commands::Man(mangen) => { + mangen + .run( + Cli::command(), + env!("CARGO_PKG_VERSION"), + env!("CARGO_PKG_NAME"), + ) + .unwrap(); + } + #[cfg(feature = "clipages")] + Commands::Md(clipages) => { + clipages.run::(env!("CARGO_PKG_NAME")).unwrap(); + } + } + + std::process::exit(0); + } } #[derive(Debug, Copy, Clone, PartialEq, Eq, Hash, Default)] diff --git a/crates/snops/src/main.rs b/crates/snops/src/main.rs index 351b119c..de3153ec 100644 --- a/crates/snops/src/main.rs +++ b/crates/snops/src/main.rs @@ -53,6 +53,9 @@ async fn main() { .try_init() .unwrap(); + // For documentation purposes will exit after running the command. + #[cfg(any(feature = "clipages", feature = "mangen"))] + Cli::parse().run(); let cli = Cli::parse(); server::start(cli).await.expect("start server"); diff --git a/crates/xtask/Cargo.toml b/crates/xtask/Cargo.toml new file mode 100644 index 00000000..5833a6f5 --- /dev/null +++ b/crates/xtask/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "xtask" +version = "0.1.0" +edition = "2021" + +# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html + +[dependencies] +anyhow.workspace = true +xshell = "0.2" diff --git a/crates/xtask/src/main.rs b/crates/xtask/src/main.rs new file mode 100644 index 00000000..55b4395d --- /dev/null +++ b/crates/xtask/src/main.rs @@ -0,0 +1,65 @@ +#![allow(dead_code)] + +use std::{env, process::Command}; + +use anyhow::{bail, Context, Result}; +use xshell::{cmd, Shell}; + +fn main() { + if let Err(e) = try_main() { + eprintln!("{e}"); + std::process::exit(1); + } +} + +const TASKS: &[&str] = &["help", "clipages", "manpages"]; + +fn try_main() -> Result<()> { + // Ensure our working directory is the toplevel + { + let toplevel_path = Command::new("git") + .args(["rev-parse", "--show-toplevel"]) + .output() + .context("Invoking git rev-parse")?; + if !toplevel_path.status.success() { + bail!("Failed to invoke git rev-parse"); + } + let path = String::from_utf8(toplevel_path.stdout)?; + std::env::set_current_dir(path.trim()).context("Changing to toplevel")?; + } + + let task = env::args().nth(1); + let sh = Shell::new()?; + match task.as_deref() { + Some("help") => print_help()?, + Some("clipages") => clipages(&sh)?, + Some("manpages") => manpages(&sh)?, + _ => print_help()?, + } + + Ok(()) +} + +fn clipages(sh: &Shell) -> Result<()> { + cmd!(sh, "cargo run -p snarkos-aot --features=docpages -- md").run()?; + cmd!(sh, "cargo run -p snops --features=docpages -- md").run()?; + cmd!(sh, "cargo run -p snops-agent --features=docpages -- md").run()?; + cmd!(sh, "cargo run -p snops-cli --features=docpages -- md").run()?; + Ok(()) +} + +fn manpages(sh: &Shell) -> Result<()> { + cmd!(sh, "cargo run -p snarkos-aot --features=docpages -- man").run()?; + cmd!(sh, "cargo run -p snops --features=docpages -- man").run()?; + cmd!(sh, "cargo run -p snops-agent --features=docpages -- man").run()?; + cmd!(sh, "cargo run -p snops-cli --features=docpages -- man").run()?; + Ok(()) +} + +fn print_help() -> Result<()> { + println!("Tasks:"); + for name in TASKS { + println!(" - {name}"); + } + Ok(()) +} diff --git a/scripts/control_plane.sh b/scripts/control_plane.sh index 0a8c6b93..21b12a65 100755 --- a/scripts/control_plane.sh +++ b/scripts/control_plane.sh @@ -1,3 +1,3 @@ #!/usr/bin/env bash -cargo run -p snops +cargo run --profile release-big -p snops diff --git a/snops_book/SUMMARY.md b/snops_book/SUMMARY.md new file mode 100644 index 00000000..085d3048 --- /dev/null +++ b/snops_book/SUMMARY.md @@ -0,0 +1,39 @@ +# Summary + +[Introduction](../README.md) + +# Architecture + +- [Overview](architecture/README.md) + - [Control Plane](architecture/CONTROL_PLANE.md) + - [Agents](architecture/AGENTS.md) + - [Runners](architecture/RUNNERS.md) + +# User Guide + +- [Overview](user_guide/README.md) + - [Setting up Environments](user_guide/envs/README.md) + - [Storage](user_guide/envs/STORAGE.md) + - [Topology](user_guide/envs/TOPOLOGY.md) + - [Cannons](user_guide/envs/CANNONS.md) + - [Timelines](user_guide/envs/TIMELINES.md) + - [Outcomes](user_guide/envs/OUTCOMES.md) + - [Running](user_guide/running/README.md) + - [Agent](user_guide/running/AGENT.md) + - [Control Plane](user_guide/running/CONTROL_PLANE.md) + - [Metrics and Logging](user_guide/running/METRICS_AND_LOGGING.md) + - [CLI Help](user_guide/clis/README.md) + - [AOT](user_guide/clis/SNARKOS_AOT.md) + - [AGENT](user_guide/clis/SNOPS_AGENT.md) + - [SNOPS](user_guide/clis/SNOPS.md) + - [SNOPS_CLI](user_guide/clis/SNOPS_CLI.md) + +# Developing + +- [Developing](developing/README.md) + +# Glossary + +- [Retention Rules](glossary/RETENTION_RULES.md) +- [Node Targets](glossary/NODE_TARGETS.md) +- [Fire Rates](glossary/FIRE_RATE.md) diff --git a/snops_book/architecture/AGENTS.md b/snops_book/architecture/AGENTS.md new file mode 100644 index 00000000..a42ec48d --- /dev/null +++ b/snops_book/architecture/AGENTS.md @@ -0,0 +1,14 @@ +# Agents + +Machines that run the `snops-agent` crate. + +Communicates with the control plane to receive state reconciliations and other messages. + +Agents are in charge of: + +- Running a [node](./RUNNERS.md). + - Validators + - Clients + - Provers +- Being a compute. For example generating transactions. +- Send off transactions. \ No newline at end of file diff --git a/snops_book/architecture/CONTROL_PLANE.md b/snops_book/architecture/CONTROL_PLANE.md new file mode 100644 index 00000000..b01c64b6 --- /dev/null +++ b/snops_book/architecture/CONTROL_PLANE.md @@ -0,0 +1,97 @@ +# Control plane: + +A machine that is running the `snops` crate. + +The control plane runs as a daemon. It orchestrates any agents that connect to +it, and listens for requests from implementors of its HTTP API (such as the +[snops-cli](TODO)) for further instructions, like preparing an +environment. + +## Responsibilites + +The control plane has many responsibilites. + +### Binary Distribution + +The control plane will manage multiple binaries: + +#### Agent Binary + +The agent binary itself. + +This allows the agents and the control plane to always be the same version. + +#### Runner Binaries + +The runner binaries. So an agent can ask for whatever version it needs from the control plane. + +### Environments + +An environment is a collection of snops documents that describe a particular use +case's storage, topology, timeline, and expected outcomes. You will learn more about Environments later in [Environments](../user_guide/envs/README.md). + +#### Topology + +The topology of agents and how they connect as validators, clients, provers, or transaction cannons. Additionally, says which storage is being used. + +#### Storage Management + +The control plane will manage the storage and generation of Ledgers for agents to ask for. + +This can be: + +- An existing ledger. +- A new ledger. +- Then genesis block of a running network, i.e. testnet. +- Ledger checkpoints. + +#### Cannons + +Cannons are transaction cannons. + +Where for public transactions you can generate them Ahead-of-Time(AOT). However, you can also generate them live. + +For private transactions only live mode is supported. + +Currently, only calls to `credits.aleo` are supported. + +#### Timelines + +An optional description of the _events_ that will be simulated. +Used to trigger actions like intentional outages, ledger manipulations, config +changes, and more. + +This is the primary vector by which snops can be used as a +testing platform. + +You can apply more than one timeline to an environment. + +#### Outcomes + +An optional description of expected outcomes after the execution of a timeline. This does requires a Prometheus instance for [metrics](#metrics-and-logging) to be running. + +For some tests, such as guaranteeing a particular TPS after some events have been simulated, it is useful to objectively verify whether or not a test succeeded. + +Outcomes are based on PromQL queries, and a Prometheus instance will be queried to check whether or not an environment's outcomes +were properly met. + +### Agent Delegation + +In the default state, the control plane holds a pool of available agents that +have connected to it. + +Agents have two States: + +- _Inventoried_: An agent is in inventory mode if it is not currently running a snarkOS node. +- _Associated_: It becomes associated with an **environment** when one is prepared. As the control plane will delegate agents in inventory to the **environment**. + +### Metrics and Logging + +For metrics and outcome guarantees, the control plane can also be linked to a +Prometheus server. The Prometheus server can be local or remote. + +For logging, the control plane can be liked to a Loki instance. + +The Prometheus metrics can also be hooked into a Grafana dashboard. The Loki instance can also be linked to a Grafana dashboard and visuzlized there. + +For an example deployment of Prometheus, Loki, and Grafana you can refer to our example configurations in `scripts/metrics`. diff --git a/snops_book/architecture/README.md b/snops_book/architecture/README.md new file mode 100644 index 00000000..a1e918ce --- /dev/null +++ b/snops_book/architecture/README.md @@ -0,0 +1,13 @@ +# Architecture + +A snops "instance" is composed of multiple parts: + +- A Control Plane +- Agents +- A runner which can be either: + + - the official snarkos runner: [snarkOS](https://github.com/AleoNet/snarkOS) + - snarkos-aot: our own modified runner and aot wrapper around `snarkOS`. + + + + + For local development, it is handy to use the agent script in the + [scripts](./scripts/) directory: + + ```bash + # start four (indexed) agents + ./scripts/agent.sh 0 + ./scripts/agent.sh 1 + ./scripts/agent.sh 2 + ./scripts/agent.sh 3 + ``` + + ### Preparing an environment + + An environment can be prepared with `snops-cli`: + + ```bash + snops-cli env prepare my-env-spec.yaml + ``` + + In a dev environment, you can use the following script: + + ```bash + ./scripts/env_start.sh specs/test-4-validators.yaml + ``` + + ### Starting the control plane + + To start the control plane, build and execute the `snops` crate binary or use + + ```bash + ./scripts/control_plane.sh + ``` -->