Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle private repositories dependencies #31

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Handle private repositories dependencies #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2ccf279
feat: :sparkles: add private repositories support (#1)
rvignolo May 10, 2024
edc6ed2
feat: ✨ add support for multiple private repos (#2)
rvignolo May 16, 2024
fb75097
feat: :construction: try deploy keys as unique var (#3)
rvignolo May 17, 2024
eba5357
Merge branch 'multiversx:main' into main
rvignolo May 17, 2024
f59e983
fix: :art: format desc (#4)
rvignolo May 17, 2024
94294c5
fix: :bug: no need to install rust if no private dependencies (#5)
rvignolo May 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 56 additions & 24 deletions .github/workflows/contracts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,59 +4,62 @@ on:
workflow_call:
inputs:
rust-toolchain:
description: 'Rust toolchain to use'
default: 'nightly'
description: "Rust toolchain to use"
default: "nightly"
required: false
type: string
pip-mxpy-args:
description: 'pip mxpy install arguments'
default: 'multiversx-sdk-cli==v9.5.2'
description: "pip mxpy install arguments"
default: "multiversx-sdk-cli==v9.5.2"
required: false
type: string
sc-meta-version:
description: 'multiversx-sc-meta version'
default: ''
description: "multiversx-sc-meta version"
default: ""
required: false
type: string
mx-scenario-go-version:
description: 'sc-scenario-go version'
default: ''
description: "sc-scenario-go version"
default: ""
required: false
type: string
path-to-sc-meta:
description: 'multiversx-sc-meta from local'
default: ''
description: "multiversx-sc-meta from local"
default: ""
required: false
type: string
clippy-args:
description: 'cargo clippy arguments'
default: '--all-targets --all-features'
description: "cargo clippy arguments"
default: "--all-targets --all-features"
required: false
type: string
enable-contracts-size-report:
description: 'Enable contracts size report'
description: "Enable contracts size report"
default: true
required: false
type: boolean
coverage-args:
description: 'sc-meta test-coverage arguments'
default: '--output ./coverage.md'
description: "sc-meta test-coverage arguments"
default: "--output ./coverage.md"
required: false
type: string
binaryen-version:
description: 'binaryen (wasm-opt) version to use'
default: 'version_112'
description: "binaryen (wasm-opt) version to use"
default: "version_112"
required: false
type: string
wabt-version:
description: 'wabt version to use'
default: '1.0.27-1'
description: "wabt version to use"
default: "1.0.27-1"
required: false
type: string
secrets:
token:
description: 'Github token'
description: "Github token"
required: true
deploy-keys:
description: "Deploy SSH private keys"
required: false

jobs:
wasm_test:
Expand All @@ -65,6 +68,14 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
uses: webfactory/[email protected]
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

Comment on lines +71 to +78
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd like to avoid using an additional third party step, if possible (and only have it as a last resort).

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand Down Expand Up @@ -110,17 +121,20 @@ jobs:
- name: Build the wasm contracts
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reviewers: https://doc.rust-lang.org/nightly/cargo/reference/config.html#netgit-fetch-with-cli

run: sc-meta all build --no-imports --target-dir $(pwd)/target --path .

- name: Run the wasm tests
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: cargo test --features multiversx-sc-scenario/run-go-tests

- name: Generate the contract report
if: ${{ inputs.enable-contracts-size-report }}
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: |
sc-meta all build-dbg --twiggy-paths --target-dir $(pwd)/target --path .
mxpy contract report --skip-build --skip-twiggy --output-format json --output-file report.json
Expand Down Expand Up @@ -178,8 +192,8 @@ jobs:
if: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && inputs.enable-contracts-size-report }}
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: 'Contract comparison'
comment-author: "github-actions[bot]"
body-includes: "Contract comparison"

- name: Create or update the report comment
uses: peter-evans/create-or-update-comment@v2
Expand All @@ -196,6 +210,14 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For contracts.yml (not for reproducible-build.yml), the solution you've previously provided (without altering contracts.yml) seems sufficient (an additional step in the workflow file of the repository itself):

multiversx/mx-sdk-rust-contract-builder#59 (comment)

Maybe we can promote and document that solution, and have contracts.yml untouched (at least, from the perspective of setting up credentials)?

uses: webfactory/[email protected]
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand Down Expand Up @@ -224,6 +246,7 @@ jobs:
- name: Run tests and generate report
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: |
sc-meta test-coverage ${{ inputs.coverage-args }}

Expand Down Expand Up @@ -256,15 +279,24 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
uses: webfactory/[email protected]
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: ${{ inputs.rust-toolchain }}

- name: Run the rust tests
env:
RUSTFLAGS: ""
run:
cargo test
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: cargo test

clippy_check:
name: Clippy linter check
Expand Down
54 changes: 47 additions & 7 deletions .github/workflows/reproducible-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
description: Image multiversx/sdk-rust-contract-builder
project_path:
type: string
default: '.'
default: "."
required: false
description: A specific project path
contract_name:
Expand All @@ -27,7 +27,11 @@ on:
description: Skip preliminary checks. Never set this in production!
package_whole_project_src:
type: boolean
description: Include all project files in the packaged source (*.source.json)
description: Include all project files in the packaged source (*.source.json)
secrets:
deploy-keys:
description: "Deploy SSH private keys"
required: false

jobs:
build:
Expand Down Expand Up @@ -86,6 +90,42 @@ jobs:
fetch-depth: 0
repository: ${{ env.GITHUB_REPOSITORY }}

- name: Setup Credentials
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We understand the need of a closed-source contract to depend on private dependencies, e.g. for development phase 👍 - of course, this is not wise for contracts deployed on mainnet - such contracts cannot pass the verified and reproducible build check.

Though, how would you feel about the options below?

(A) have your private dependencies brought as git submodules in your main private repository, then alter the Cargo.toml to point to them (of course, making your project open-source would reveal the source code of the dependencies, as well).

(B) in the Docker images for reproducible builds, we can enable ssh-agent, and run ssh-add (if any key is provided in the Docker environment) - I did not yet test this approach, though.

(C) allow reproducible-build.yml to be parametrized with the fully qualified-name of a Docker image (instead of image_tag only) - this way, you can use a custom Docker image to perform the build of a closed-source contract.

uses: webfactory/[email protected]
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
toolchain: ${{ inputs.rust-toolchain }}
target: wasm32-unknown-unknown

- name: Vendored dependencies (if private dependencies are used)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thus, private dependencies are brought locally, before starting the build, correct (sorry if I misunderstood)?

If so, the usual contract verification flow (which relies on the Docker container to perform the whole build) would not be able to cover this case (however, for verified contracts, private dependencies would not be acceptable, anyway).

env:
CARGO_NET_GIT_FETCH_WITH_CLI: true
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
run: |
mkdir .cargo
shopt -s globstar
cargos=($(echo **/Cargo.toml))
cargos_s=$(printf -- '-s %s ' "${cargos[@]}")
cargo vendor ${cargos_s} > .cargo/config.toml

# Prevent mx-sdk-rust-contract-builder from bulding vendored dependencies
rm -f vendor/**/multiversx.json

for checksum_file in **/.cargo-checksum.json; do
jq 'del(.files."multiversx.json")' "$checksum_file" > "$checksum_file.tmp" && mv "$checksum_file.tmp" "$checksum_file"
done

- name: Preliminary checks
if: ${{ inputs.skip_preliminary_checks == false }}
run: |
Expand All @@ -110,7 +150,7 @@ jobs:
else:
logging.error(f"wasm/Cargo.lock file not found: {cargo_lock}")
missing_cargo_lock = True

if missing_cargo_lock:
sys.exit(f"ERROR: One or more 'wasm/Cargo.lock' files are missing. They are essential for reproducible builds.")
EOF
Expand All @@ -120,16 +160,16 @@ jobs:
- name: Download build script
run: |
wget https://raw.githubusercontent.com/multiversx/mx-sdk-rust-contract-builder/${{ inputs.image_tag }}/build_with_docker.py

- name: Build contracts
run: |
flag_package_whole_project_src=""
if ${{ inputs.package_whole_project_src }}; then
flag_package_whole_project_src="--package-whole-project-src"
fi

python3 ./build_with_docker.py --no-docker-tty --image=multiversx/sdk-rust-contract-builder:${{ inputs.image_tag }} --project=${{ inputs.project_path }} --contract=${{ inputs.contract_name }} --output=/home/runner/work/output-from-docker ${flag_package_whole_project_src}

- name: Save artifacts
uses: actions/upload-artifact@v3
with:
Expand Down Expand Up @@ -167,7 +207,7 @@ jobs:

try:
logging.info(f"Fetching existing release notes, if any, for repository = {repository}, tag = {ref_name} ...")

request = urllib.request.urlopen(f"https://api.github.com/repos/{repository}/releases/tags/{ref_name}")
data = json.loads(request.read())
existing_notes = data["body"]
Expand Down