diff --git a/lib/docs/docs/architecture.md b/lib/docs/docs/architecture.md new file mode 100644 index 0000000..c9bb3b6 --- /dev/null +++ b/lib/docs/docs/architecture.md @@ -0,0 +1,68 @@ +# πŸ—ΊοΈ Architecture details + +This page presents the project architecture and some technical details. + +## πŸ”οΈ Changelog + +Version history is recorded in the [CHANGELOG.md](https://github.com/vemonet/nanopub-rs/blob/main/CHANGELOG.md). + +## πŸ—ƒοΈ Folder structure + +``` +nanopub-rs/ +β”œβ”€β”€ lib/ +β”‚ β”œβ”€β”€ src/ +β”‚ β”‚ └── πŸ¦€ Source code for the core Rust crate. +β”‚ β”œβ”€β”€ tests/ +β”‚ β”‚ └── πŸ§ͺ Tests for the core Rust crate. +β”‚ └── docs/ +β”‚ └── πŸ“– Markdown and HTML files for the documentation website. +β”œβ”€β”€ python/ +β”‚ └── 🐍 Python bindings for interacting with the Rust crate. +β”œβ”€β”€ js/ +β”‚ └── 🌐 JavaScript bindings for integrating into JS environments. +β”œβ”€β”€ cli/ +β”‚ └── ⌨️ Scripts for the command-line interface. +β”œβ”€β”€ scripts/ +β”‚ └── πŸ› οΈ Development scripts (build docs, testing). +└── .github/ + └── workflows/ + └── βš™οΈ Automated CI/CD workflows. +``` + +## βœ’οΈ Nanopub signing process + +1. Preliminary nanopub is created with blank space in URIs at the places where the trusty URI code will appear (normalized URI: `https://w3id.org/np/ `, cf. [original java implementation](https://github.com/Nanopublication/nanopub-java/blob/22bba0e79508309f1c6163970f49ab596beadeb0/src/main/java/org/nanopub/trusty/TempUriReplacer.java#L12)); this includes the signature part, except the triple that is stating the actual signature +2. Preliminary nanopub is serialized in a normalized fashion (basically each quad on four lines with minimal escaping) +3. Signature is calculated on this normalized representation (cf. most of the process in the [trusty-uri python lib](https://github.dev/trustyuri/trustyuri-python/blob/9f29732c4abae9d630d36e6da24720e02f543ebf/trustyuri/rdf/RdfHasher.py#L15), see also [SignatureUtils](https://github.com/Nanopublication/nanopub-java/blob/22bba0e79508309f1c6163970f49ab596beadeb0/src/main/java/org/nanopub/extra/security/SignatureUtils.java#L196) and [trusty-uri](https://github.com/trustyuri/trustyuri-java/blob/08b61fbb13d20a5cbefde617bd9a9e9b0b03d780/src/main/java/net/trustyuri/rdf/RdfHasher.java#L86)) +4. Signature triple is added +5. Trusty URI code is calculated on normalized representation that includes signature +6. Trusty URI code is added in place of all the occurrences of blank spaces in the URIs, leading to the final trusty nanopub + +## πŸ› οΈ Notes about maintenance and stability + +Cross-compiling to many targets brings some complexity to the build process, especially that the nanopub lib packs a lot of features: processing RDF, RSA signing and random key generation, querying a HTTP server, getting current datetime. + +This means we need to make sure the dependencies we use work for all compilation targets (e.g. aarch64, wasm). And in some case we need to define platform dependant dependencies in the `Cargo.toml` (e.g. reqwest `native-tls` for aarch64 windows instead of the `rustls-tls`) + +Packages are built for different targets in the `.github/workflows/build.yml` GitHub action. + +## β˜‘οΈ To do + +- [ ] Add possibility to build the nanopub from scratch for JS and python +- [ ] Integrate to the python `nanopub` library to perform signing? +- [ ] Add Ruby bindings? https://docs.rs/magnus/latest/magnus https://github.com/ankane/tokenizers-ruby +- [ ] Add Java bindings? https://docs.rs/jni/latest/jni +- [ ] Add brew packaging? (cf. [ripgrep](https://github.com/BurntSushi/ripgrep/blob/master/pkg/brew/ripgrep-bin.rb)) + +## ⏱️ Speed comparison + +Speed taken when signing a nanopub using different languages implementations (in this order: [java](https://github.com/Nanopublication/nanopub-java), [python](https://github.com/fair-workflows/nanopub), rust): + +| Command | Mean [ms] | Min [ms] | Max [ms] | Relative | +|:---|---:|---:|---:|---:| +| `java -jar nanopub.jar sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 319.5 Β± 11.2 | 296.0 | 337.5 | 60.74 Β± 2.49 | +| `np sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 446.6 Β± 3.2 | 441.2 | 457.6 | 84.93 Β± 1.93 | +| `target/release/nanopub-cli sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 5.3 Β± 0.1 | 5.1 | 6.3 | 1.00 | + +> Tested in GitHub actions on Ubuntu. diff --git a/lib/docs/docs/contributing.md b/lib/docs/docs/contributing.md index 4cbb189..fe7ea97 100644 --- a/lib/docs/docs/contributing.md +++ b/lib/docs/docs/contributing.md @@ -1,4 +1,4 @@ -# πŸ› οΈ Contributing +# πŸ› οΈ Development [![Build](https://github.com/vemonet/nanopub-rs/actions/workflows/build.yml/badge.svg)](https://github.com/vemonet/nanopub-rs/actions/workflows/build.yml) [![Lint and Test](https://github.com/vemonet/nanopub-rs/actions/workflows/test.yml/badge.svg)](https://github.com/vemonet/nanopub-rs/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/vemonet/nanopub-rs/graph/badge.svg?token=BF15PSO6GN)](https://codecov.io/gh/vemonet/nanopub-rs) [![dependency status](https://deps.rs/repo/github/vemonet/nanopub-rs/status.svg)](https://deps.rs/repo/github/vemonet/nanopub-rs) @@ -9,60 +9,20 @@ The usual process to make a contribution is to: 3. Make your changes 4. Make sure formatting, linting and tests passes. 5. Add tests if possible to cover the lines you added. -6. Commit, and send a Pull Request. +6. [Commit](https://www.conventionalcommits.org/en/v1.0.0/), and send a Pull Request. -## οΈπŸ—ΊοΈ Architecture details +## πŸ“₯️ Clone the repository -### πŸ—ƒοΈ Folder structure +Clone the `nanopub-rs` repository, and `cd` into it: +```bash +git clone https://github.com/vemonet/nanopub-rs.git +cd nanopub-rs ``` -nanopub-rs/ -β”œβ”€β”€ lib/ -β”‚ β”œβ”€β”€ src/ -β”‚ β”‚ └── πŸ¦€ Source code for the core Rust crate. -β”‚ β”œβ”€β”€ tests/ -β”‚ β”‚ └── πŸ§ͺ Tests for the core Rust crate. -β”‚ └── docs/ -β”‚ └── πŸ“– Markdown and HTML files for the documentation website. -β”œβ”€β”€ python/ -β”‚ └── 🐍 Python bindings for interacting with the Rust crate. -β”œβ”€β”€ js/ -β”‚ └── 🌐 JavaScript bindings for integrating into JS environments. -β”œβ”€β”€ cli/ -β”‚ └── ⌨️ Scripts for the command-line interface. -β”œβ”€β”€ scripts/ -β”‚ └── πŸ› οΈ Development scripts (build docs, testing). -└── .github/ - └── workflows/ - └── βš™οΈ Automated CI/CD workflows. -``` - -### βœ’οΈ Nanopub signing process - -- Preliminary nanopub is created with blank space in URIs at the places where the trusty URI code will appear (normalized URI: `https://w3id.org/np/ `, cf. [original java implementation](https://github.com/Nanopublication/nanopub-java/blob/22bba0e79508309f1c6163970f49ab596beadeb0/src/main/java/org/nanopub/trusty/TempUriReplacer.java#L12)); this includes the signature part, except the triple that is stating the actual signature -- Preliminary nanopub is serialized in a normalized fashion (basically each quad on four lines with minimal escaping) -- Signature is calculated on this normalized representation (cf. most of the process in the [trusty-uri python lib](https://github.dev/trustyuri/trustyuri-python/blob/9f29732c4abae9d630d36e6da24720e02f543ebf/trustyuri/rdf/RdfHasher.py#L15), see also [SignatureUtils](https://github.com/Nanopublication/nanopub-java/blob/22bba0e79508309f1c6163970f49ab596beadeb0/src/main/java/org/nanopub/extra/security/SignatureUtils.java#L196) and [trusty-uri](https://github.com/trustyuri/trustyuri-java/blob/08b61fbb13d20a5cbefde617bd9a9e9b0b03d780/src/main/java/net/trustyuri/rdf/RdfHasher.java#L86)) -- Signature triple is added -- Trusty URI code is calculated on normalized representation that includes signature -- Trusty URI code is added in place of all the occurrences of blank spaces in the URIs, leading to the final trusty nanopub - -### πŸ› οΈ Notes about maintenance and stability - -Cross-compiling to many targets brings some complexity to the build process, especially that the nanopub lib packs a lot of features: processing RDF, RSA signing and key generation, querying a HTTP server, getting current datetime access. - -This means we need to make sure the dependencies we use work for all compilation targets (e.g. aarch64, wasm). -### β˜‘οΈ To do +## βš™οΈ Install dependencies -- [ ] Add possibility to build the nanopub from scratch for JS and python -- [ ] Integrate to the python `nanopub` library to perform signing? -- [ ] Add Ruby bindings? https://docs.rs/magnus/latest/magnus https://github.com/ankane/tokenizers-ruby -- [ ] Add Java bindings? https://docs.rs/jni/latest/jni -- [ ] Add brew packaging? (cf. [ripgrep](https://github.com/BurntSushi/ripgrep/blob/master/pkg/brew/ripgrep-bin.rb)) - -## πŸ§‘β€πŸ’» Development workflow - -[Rust](https://www.rust-lang.org/tools/install), python, and NodeJS are required for development. +[Rust](https://www.rust-lang.org/tools/install), [Python](https://www.python.org/downloads/), and [NodeJS](https://nodejs.org/en/download) are required for development. Install development dependencies: @@ -79,19 +39,12 @@ pre-commit install # Install rust dev tools rustup update -cargo install wasm-pack cargo-tarpaulin cargo-deny git-cliff +cargo install wasm-pack cargo-tarpaulin cargo-deny cargo-outdated git-cliff ``` -### πŸ“₯️ Clone the repository - -Clone the `nanopub-rs` repository, `cd` into it, and create a new branch for your contribution: - -```bash -cd nanopub-rs -git checkout -b add-my-contribution -``` +## πŸ§ͺ Run tests -### πŸ§ͺ Test Rust crate +### πŸ¦€ Test Rust crate Run tests for the rust crate: @@ -119,7 +72,7 @@ cargo test cargo test -- --test-threads=1 ``` -Test the `nanopub` crate with code coverage: +Test the `nanopub` crate with code coverage (much slower): ```bash cargo tarpaulin -p nanopub --out html @@ -172,25 +125,29 @@ cargo run -- sign ../lib/tests/resources/nanopub_test_blank.trig ./scripts/test-all.sh ``` -### ✨ Format +## 🧼 Format & lint + +Automatically format the codebase using `rustfmt`: ```bash cargo fmt ``` -### 🧹 Lint +Lint with `clippy`: ```bash cargo clippy --all --all-targets --all-features ``` -### πŸ“– Generate docs +## πŸ“– Generate docs + +Start docs website locally with mkdocs: ```bash ./scripts/docs.sh ``` -### 🎭️ Work on the demo webpage +## 🎭️ Work on the demo webpage Start a web server at [localhost:3000/playground.html](http://localhost:3000/playground.html) @@ -198,16 +155,7 @@ Start a web server at [localhost:3000/playground.html](http://localhost:3000/pla python -m http.server 3000 --directory ./lib/docs ``` -### πŸ“¦οΈ Build and run - -All packages at once: - -```bash -cargo build --all -cargo run --all-features -``` - -### ️⛓️ Check supply chain +## ️⛓️ Check supply chain Check the dependency supply chain: licenses (only accept dependencies with OSI or FSF approved licenses), and vulnerabilities (CVE advisories). @@ -215,43 +163,23 @@ Check the dependency supply chain: licenses (only accept dependencies with OSI o cargo deny check ``` -### 🏷️ New release - -Publishing artifacts will be done by the `build.yml` workflow, make sure you have set the following tokens as secrets for this repository: `PYPI_TOKEN`, `NPM_TOKEN`, `CRATES_IO_TOKEN`, `CODECOV_TOKEN` - -Install dependency: +Make sure dependencies have been updated: ```bash -cargo install cargo-release cargo-outdated +cargo update +cargo outdated ``` -1. Make sure dependencies have been updated: +## 🏷️ Publish a new release - ```bash - cargo update - cargo outdated - ``` +Building and publishing artifacts will be done by the [`build.yml`](https://github.com/vemonet/nanopub-rs/actions/workflows/build.yml) GitHub actions workflow, make sure you have set the following tokens as secrets for this repository: `PYPI_TOKEN`, `NPM_TOKEN`, `CRATES_IO_TOKEN`, `CODECOV_TOKEN` -2. Run the release script, it will bump the version in the `Cargo.toml` files, generate the changelog, commit, create a new tag, and push to GitHub +Then just run the release script providing the new version following [semantic versioning](https://semver.org), it will bump the version in the `Cargo.toml` files, generate the changelog from commit messages, create a new tag, and push to GitHub: - ```bash - ./scripts/release.sh 0.0.2 - ``` - -4. The `build.yml` workflow will automatically build artifacts (binary, pip wheel, npm package), create a new release on GitHub, and add the artifacts to the new release. - -## ⏱️ Speed comparison - -Speed taken when signing a nanopub using different languages implementations (in this order: [java](https://github.com/Nanopublication/nanopub-java), [python](https://github.com/fair-workflows/nanopub), rust): - -| Command | Mean [ms] | Min [ms] | Max [ms] | Relative | -|:---|---:|---:|---:|---:| -| `java -jar nanopub.jar sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 319.5 Β± 11.2 | 296.0 | 337.5 | 60.74 Β± 2.49 | -| `np sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 446.6 Β± 3.2 | 441.2 | 457.6 | 84.93 Β± 1.93 | -| `target/release/nanopub-cli sign lib/tests/resources/simple1-rsa.trig -k lib/tests/resources/id_rsa` | 5.3 Β± 0.1 | 5.1 | 6.3 | 1.00 | - -> Tested in GitHub actions on Ubuntu. +```bash +./scripts/release.sh 0.0.2 +``` -## πŸ”οΈ Changelog +!!! success "Automated release" -Version history is recorded in the [CHANGELOG.md](https://github.com/vemonet/nanopub-rs/blob/main/CHANGELOG.md). + The `build.yml` workflow will automatically build artifacts (binaries, pip wheels, npm package), create a new release on GitHub, and add the generated artifacts to the new release. diff --git a/lib/docs/mkdocs.yml b/lib/docs/mkdocs.yml index 5e47b5c..629bd9e 100644 --- a/lib/docs/mkdocs.yml +++ b/lib/docs/mkdocs.yml @@ -1,5 +1,5 @@ site_name: Nanopub cross-platform toolkit -site_description: A cross-platform Rust library to sign Nanopublications, with bindings to Python and JS (wasm) +site_description: A cross-platform toolkit to sign and publish Nanopublications, written in Rust with bindings to Python and JS (wasm) site_author: Vincent Emonet site_url: https://vemonet.github.io/nanopub-rs repo_name: vemonet/nanopub-rs @@ -15,7 +15,7 @@ theme: admonition: server: material/server language: en - # https://squidfunk.github.io/mkdocs-material/setup/changing-the-colors/#primary-color + # Change color: https://squidfunk.github.io/mkdocs-material/setup/changing-the-colors/#primary-color palette: - media: "(prefers-color-scheme: light)" scheme: default @@ -32,35 +32,32 @@ theme: icon: material/weather-sunny name: Switch to light mode features: - - content.code.copy - - content.code.annotate - - content.code.select # Still experimental - # - content.tabs.link - # - header.autohide - # - navigation.expand - navigation.indexes - # - navigation.instant - navigation.sections - navigation.tabs - # - navigation.tabs.sticky - navigation.top - navigation.tracking + - content.code.copy + - content.code.annotate + - content.code.select - search.highlight - search.share - search.suggest - toc.follow + # - header.autohide + # - navigation.tabs.sticky + # - navigation.expand + # - navigation.instant + # - content.tabs.link -# https://fontawesome.com/icons/ +# Find icons: https://fontawesome.com/icons/ # https://squidfunk.github.io/mkdocs-material/reference/icons-emojis/ nav: - Docs: - Introduction: index.md - Command Line Interface: cli.md - Use from Rust: rust.md - # - Deploy: - # - 🐳 With Docker: use-docker.md - # - 🐍 With pip: use-pip.md - # - Advanced: + - Architecture details: architecture.md - Development: contributing.md - JavaScript: - Use from JavaScript: javascript.md @@ -88,7 +85,8 @@ watch: - docs markdown_extensions: - - admonition # https://squidfunk.github.io/mkdocs-material/reference/admonitions/#supported-types + - admonition + # Supported admonititions: https://squidfunk.github.io/mkdocs-material/reference/admonitions/#supported-types - pymdownx.highlight: anchor_linenums: true - pymdownx.inlinehilite @@ -114,9 +112,13 @@ extra_css: extra: social: + - icon: fontawesome/brands/rust + link: https://crates.io/crates/nanopub + - icon: fontawesome/brands/npm + link: https://www.npmjs.com/package/@nanopub/sign - icon: fontawesome/brands/python - link: https://pypi.org/project/nanopub + link: https://pypi.org/project/nanopub-sign + - icon: fontawesome/brands/github + link: https://github.com/Nanopublication # - icon: fontawesome/brands/docker # link: https://github.com/vemonet/nanopub-rs/pkgs/container/nanopub-rs - - icon: fontawesome/brands/github - link: https://github.com/vemonet diff --git a/lib/docs/requirements.txt b/lib/docs/requirements.txt index d31a7d3..89fe7bc 100644 --- a/lib/docs/requirements.txt +++ b/lib/docs/requirements.txt @@ -2,5 +2,5 @@ mkdocs >=1.4.2 mkdocs-material >=8.2.7 mkdocstrings[python] >=0.19.1 mdx-include >=1.4.1 -mkdocs-markdownextradata-plugin >=0.2.5 # ,<0.3.0 +mkdocs-markdownextradata-plugin >=0.2.5 mkdocs-open-in-new-tab diff --git a/scripts/cov.sh b/scripts/cov.sh index d47a406..137c989 100755 --- a/scripts/cov.sh +++ b/scripts/cov.sh @@ -1,6 +1,8 @@ #!/usr/bin/env bash set -e +# Run tests with coverage + cargo tarpaulin -p nanopub --out html --timeout 120 \ --exclude-files lib/src/error.rs diff --git a/scripts/docs.sh b/scripts/docs.sh index bc0c326..2aedaad 100755 --- a/scripts/docs.sh +++ b/scripts/docs.sh @@ -1,19 +1,7 @@ #!/usr/bin/env bash set -e -# source scripts/docs-build.sh - -# echo "πŸ¦€ Rust doc at http://0.0.0.0:3000/doc/nanopub" -# echo "πŸ“– MdBook at http://0.0.0.0:3000" - -# python -m http.server 3000 --directory ./target/doc - -# python3 -m webbrowser ./target/doc/ - - -# With mkdocs: - -# pip install -r lib/docs/requirements.txt +# Start mkdocs in development if [ ! -d ".venv" ]; then echo ".venv virtual environment does not exist. Creating it"