Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update nix build and local dev guides #1954

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
### Changed

- Changed an internal type to improve IntelliSense on ZkProgram methods https://github.com/o1-labs/o1js/pull/1933
- Updated build:bindings and build:update-bindings to recompile Rust on all builds

### Fixed

Expand Down
16 changes: 16 additions & 0 deletions README-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,3 +241,19 @@ To facilitate this process, use the provided script named `run-debug`. To use th
This script initializes a Node.js process with the `--inspect-brk` flag that starts the Node.js inspector and breaks before the user script starts (i.e., it pauses execution until a debugger is attached). The `--enable-source-maps` flag ensures that source maps are used to allow easy debugging of o1js code directly.

After the Node.js process is running, open the Chrome browser and navigate to `chrome://inspect` to attach the Chrome Debugger to the Node.js process. You can set breakpoints, inspect variables, and profile the performance of your zkApp or o1js. For more information on using the Chrome Debugger, see the [DevTools documentation](https://developer.chrome.com/docs/devtools/).

### Debugging within the SDK
To debug a call into the SDK, you can link your local copy of the SDK with `npm link`. After that, you'll be able to add log statements, set breakpoints, and make code changes. Within the SDK, run:
```sh
npm run link
```
Then in your zkApp codebase, run:
```sh
npm link o1js
```

#### Logging from OCaml
If you need to debug a call into the OCaml code, the process is a little more complicated. The OCaml is compiled into JavaScript with js_of_ocaml during `npm run build:update-bindings`, so you'll need to add your logs into the OCaml code and rebuild the bindings to see them. Logging from OCaml in a way that will reflect as JS `console.log`s in the compiled code can be done like this:
```ocaml
let () = print_endline "This is a log" in
```
38 changes: 21 additions & 17 deletions README-nix.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,21 +92,7 @@ nix develop o1js#default
The first time you run this command, you can expect it to take hours (or even a full day) to complete. Then, you will observe that the current devshell becomes a Nix shell with the right
configuration for `o1js` and `mina`.

In order to make sure that the bindings will be regenerated in the case that you
are modifying them, make sure to comment out the conditionals in
`src/mina/src/lib/crypto/kimchi_bindings/js/node_js/build.sh` and `src/mina/src/lib/crypto/kimchi_bindings/js/web/build.sh` locally. That's because otherwise the
PLONK_WASM_WEB check prevents `proof-systems` from compiling with each build.

```sh
if true; then # [[ -z "${PLONK_WASM_WEB-}" ]]; then
export RUSTFLAGS="-C target-feature=+atomics,+bulk-memory,+mutable-globals -C link-arg=--no-check-features -C link-arg=--max-memory=4294967296"
rustup run nightly-2023-09-01 wasm-pack build --target web --out-dir ../js/web ../../wasm -- -Z build-std=panic_abort,std
else
cp "$PLONK_WASM_WEB"/* -R .
fi
```

Then, you can build o1js and update the bindings.
From within the shell, you can build o1js and update the bindings.

```console
npm run build
Expand Down Expand Up @@ -152,7 +138,7 @@ The second flag increases the default number of jobs being 1, so that rebuilding

The last two lines tell Nix to use the Mina Foundation's cache whenever possible, which should as well speed things up when building code that has been build in Mina's CI before.

## Common errors
## Common Issues

Errors while using Nix have been reported. This section collects a set of common
errors and proposes fixes for them.
Expand Down Expand Up @@ -305,4 +291,22 @@ Then, the error message would still contain old directories.

#### Fix

Create a new environment for Nix and start from scratch. In particular, run the garbage collector which will remove old dependencies.
Rerun `pin.sh` and `src/mina/nix/pin.sh`.

### Changes to nix flakes aren't taking effect

On MacOS, nix may ignore changes to files when nix commands are run and reuse the flake cached in its registry. Running commands like `nix develop o1js` and `nix run o1js#update-bindings` will reuse the cached version of the flake. As a result:

- The devshell could be missing newly added dependencies.
- Builds executed directly with `nix run` could be generated from old source files.

#### Fix

There are two ways to ensure Nix recognizes flake changes:

- Rerun `pin.sh` to force an update to the registry, then run your command.
- Reference the flake by its directory path rather than its registry name. This forces Nix to use the current contents of the directory:

```bash
nix develop .'?submodules=1#default'
```
4 changes: 2 additions & 2 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 26 additions & 25 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -98,27 +98,28 @@
#Rustup doesn't allow local toolchains to contain 'nightly' in the name
#so the toolchain is linked with the name nix and rustup is wrapped in a shellscript
#which calls the nix toolchain instead of the nightly one
(writeShellApplication
{ name = "rustup";
text =
''
if [ "$1" = run ] && { [ "$2" = nightly-2023-09-01 ] || [ "$2" = 1.72-x86_64-unknowl-linux-gnu ]; }
then
echo using nix toolchain
${rustup}/bin/rustup run nix "''${@:3}"
else
echo using plain rustup "$@"
${rustup}/bin/rustup "$@"
fi
'';
}
)
# (writeShellApplication
# { name = "rustup";
# checkPhase = with pkgs; !stdenv.isDarwin;
# text =
# ''
# if [ "$1" = run ] && { [ "$2" = nightly-2023-09-01 ] || [ "$2" = 1.72-x86_64-unknowl-linux-gnu ]; }
# then
# echo using nix toolchain
# ${rustup}/bin/rustup run nix "''${@:3}"
# else
# echo using plain rustup "$@"
# ${rustup}/bin/rustup "$@"
# fi
# '';
# }
# )
rustup
wasm-pack
binaryen # provides wasm-opt

dune_3
] ++ commonOverrides.buildInputs ;
] ++ commonOverrides.buildInputs;

inherit (nixpkgs) lib;
# All the submodules required by .gitmodules
Expand Down Expand Up @@ -204,12 +205,12 @@
# This seems to work better for macos
mina-shell = requireSubmodules inputs.mina.devShells."${system}".with-lsp;
default = requireSubmodules (pkgs.mkShell {
shellHook =
''
RUSTUP_HOME=$(pwd)/.rustup
export RUSTUP_HOME
rustup toolchain link nix ${rust-channel}
'';
# shellHook =
# ''
# RUSTUP_HOME=$(pwd)/.rustup
# export RUSTUP_HOME
# rustup toolchain link nix ${rust-channel}
# '';
packages = bindings-pkgs;
});

Expand Down Expand Up @@ -245,18 +246,18 @@
];
});
inherit (inputs.mina.devShells."${system}".default)
PLONK_WASM_NODEJS
PLONK_WASM_WEB
MARLIN_PLONK_STUBS
;
PREBUILT_KIMCHI_BINDINGS_JS_WEB =
"${mina.files.src-lib-crypto-kimchi_bindings-js-web}/src/lib/crypto/kimchi_bindings/js/web";
PREBUILT_KIMCHI_BINDINGS_JS_NODE_JS =
"${mina.files.src-lib-crypto-kimchi_bindings-js-node_js}/src/lib/crypto/kimchi_bindings/js/node_js";
EXPORT_TEST_VECTORS = "${test-vectors}/bin/export_test_vectors";
buildInputs = bindings-pkgs ++ [ pkgs.bash ];
SKIP_MINA_COMMIT = true;
JUST_BINDINGS = true;

buildInputs = bindings-pkgs ++ [ pkgs.bash ];

patchPhase = ''
patchShebangs ./src/bindings/scripts/
patchShebangs ./src/bindings/crypto/test-vectors/
Expand Down
2 changes: 1 addition & 1 deletion src/mina
Submodule mina updated 682 files
Loading