Feat: Vm Profiling #1789

nhtyy · 2024-11-14T00:44:55Z

Building on a previous PR #1713

The logic to preserve the callstack is now done during execution (debug only)
and by default the trace file is now a gecko profile json, which can opened via samply load ...

yuwen01 · 2024-11-14T18:11:30Z

book/developers/profiling.md

@@ -0,0 +1,24 @@
+# Profiling ZKVM programs
+


We can either replace the ## Tracking Cycles With Tracing section in book/writing-programs/cycle-tracking.md, or we can move the ## Tracking Cycles With Annotations section from that file here. Either way, we're trying to deprecate the old cargo prove trace thing right?

Oh I didnt see that thanks

yuwen01

overall looks pretty good, made some small comments about cleaning up docs

yuwen01 · 2024-11-14T18:14:50Z

book/developers/profiling.md

+To enable profiling, set the `TRACE_FILE` env var to the path where you want the profile to be saved.
+
+The full command to profile should look something like this
+```sh


as discussed irl: either add RUST_FLAGS or, if the tracing doesn't impact runtime performance too much, get rid of the debug assertions thing

RUST_FLAGS doesnt seem to be an option afaict since debug_assertions seems to be owned by cargo

yuwen01 · 2024-11-14T18:18:30Z

crates/core/executor/src/profiler.rs

+        Ok(())
+    }
+
+    /// Simple Check to makes sure we have valid main function that lasts for most of the exeuction


crates/core/executor/src/profiler.rs

nhtyy · 2024-11-14T22:54:17Z

@yuwen01 the CLI CI test seems to be failing silently here.. Wdyt?

yuwen01

Left some doc comments, but logic is pretty clear now thanks to your comments. You can check that your changes to the book look good by running mdbook serve from the sp1 root directory

yuwen01 · 2024-11-14T18:30:34Z

book/developers/profiling.md

+
+Profiling the ZKVM can only be done with debug builds, and special care must be taken to ensure correctness, only one program may be profiled at a time.
+
+To profile a program, you have to setup a script to execute the program, many examples can be found in the repo, such as this ('fibonacci')[../../examples/fibonacci/script/src/main.rs] script.


link broken

yuwen01 · 2024-11-14T23:52:50Z

book/writing-programs/cycle-tracking.md

-
-```bash
-TRACE_FILE=trace.log RUST_LOG=info cargo run --release
+To profile a program, you have to setup a script to execute the program, many examples can be found in the repo, such as this ('fibonacci')[../../examples/fibonacci/script/src/main.rs] script.


To profile a program, you have to setup a script to execute the program. Many examples can be found in the repo, such as this ['fibonacci'](../../examples/fibonacci/script/src/main.rs) script.

yuwen01 · 2024-11-14T23:53:23Z

book/writing-programs/cycle-tracking.md


-The `cycle-tracker` annotation is a convenient way to track cycles for specific sections of code. However, sometimes it can also be useful to track what functions are taking the most cycles across the entire program, without having to annotate every function individually.
+Profiling the VM is a good way to get an understanding of what is bottlenecking your program, and special care must be taken to ensure correctness, only one program may be profiled at a time.


Profiling the zkVM is a good way to get an understanding of what is bottlenecking your program. Note that only one program may be profiled at a time.

yuwen01 · 2024-11-14T23:54:10Z

book/writing-programs/cycle-tracking.md

-When the `TRACE_FILE` environment variable is set, as SP1's RISC-V runtime is executing, it will write a log of the program counter to the file specified by `TRACE_FILE`.
-
-Next, we can use the `cargo prove` CLI with the `trace` command to analyze the trace file and generate a table of instruction counts. This can be done with the following command:
+The data captured by the profiler can be quite large, you can set the sample rate using the `TRACE_SAMPLE_RATE` env var.


Can you explain that a higher sample rate will result in smaller files? Someone might think that a small rate will decrease the sampling frequency

yuwen01 · 2024-11-14T23:54:38Z

book/writing-programs/cycle-tracking.md

-| syscall_hint_read                      | 3                 |
-+----------------------------------------+-------------------+
+
+To profile in release mode you can use the following setup in your `Cargo.toml`


isn't this no longer necessary

yuwen01 · 2024-11-14T23:55:57Z

crates/core/executor/src/profiler.rs

+                let start_address = sym.st_value;
+                let end_address = start_address + size - 4;
+
+                // now that we have the name lets immeidalty intern it so we only need to copy


typo, also you should use complete sentences and capitalization.

yuwen01 · 2024-11-14T23:59:04Z

crates/core/executor/src/profiler.rs

+
+/// The ZKVM Profiler.
+///
+/// During exeuction, the profiler always keeps track of the callstack


exeuction -> execution

yuwen01 · 2024-11-15T00:00:34Z

book/writing-programs/cycle-tracking.md

 ```

-The `trace` command will generate a table of instruction counts, sorted by the number of cycles spent in each function. The output will look something like this:
-
+To view these profiles, we recommend (Samply)[https://github.com/mstange/samply].


broken link

gp

nhtyy force-pushed the n/profiling-in-executor branch 2 times, most recently from e04ad2b to 5e1c1b9 Compare November 14, 2024 00:52

yuwen01 reviewed Nov 14, 2024

View reviewed changes

nhtyy force-pushed the n/profiling-in-executor branch from 27923bd to a1a732c Compare November 14, 2024 18:22

yuwen01 reviewed Nov 14, 2024

View reviewed changes

nhtyy mentioned this pull request Nov 14, 2024

chore: deprecate cargo prove #1792

Open

nhtyy force-pushed the n/profiling-in-executor branch 2 times, most recently from 9c5d7fc to 40b61fc Compare November 14, 2024 23:28

yuwen01 reviewed Nov 15, 2024

View reviewed changes

nhtyy force-pushed the n/profiling-in-executor branch from 2af35b6 to b5a3684 Compare November 16, 2024 00:19

nhtyy and others added 19 commits November 15, 2024 16:24

feat(cli): profile

722808d

fix: unused code

3ccf0c7

feat: sample rate

21bd170

chore: remove old tracing cmd, samply load

a45a2fa

chore: sample validity check, gecko from cratesio

d3ef00b

feat(executor): sample rate

1fd057b

chore: License

43e11c5

chore: comments

dc22507

chore: cleanup, err context

ab5793f

fix: clk in log

7d57a17

fix

f83b616

add"

34eabd1

gp

better profiling performance

dca235b

fix: clippy

5c29252

fix: only set sample rate at trace time

53cd185

chore: progress bar during sampling

a7e56de

feat: profiling logic in executor, docs, rm profile cli cmd

af3ad9f

fix: docs samply

602b66b

fix: resotre test, unused dep in cli

0015185

nhtyy added 14 commits November 15, 2024 16:24

fix: fmt + clippy, unused util

808d697

fix: spelling

34ee98c

perf: intern strings first

b000575

fix: ci

b859cff

fix: feature flags

e180918

fix: doc comments, release mode fix

1e78558

chore: rm debug assertions for profiler

9beec03

fix: must use clk not pc for sampling rate

760920c

fix: caps

1555f6c

fix: prover no debug assertions on executor

0b661b3

fix: fmt

e1f4804

fix: use global clk

1904003

fix: book + typos

b5d5c79

unused cli dep

6b834d1

nhtyy force-pushed the n/profiling-in-executor branch from b5a3684 to 6b834d1 Compare November 16, 2024 00:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Vm Profiling #1789

Feat: Vm Profiling #1789

nhtyy commented Nov 14, 2024

yuwen01 Nov 14, 2024 •

edited

Loading

nhtyy Nov 14, 2024

yuwen01 left a comment

yuwen01 Nov 14, 2024

nhtyy Nov 14, 2024

yuwen01 Nov 14, 2024

nhtyy commented Nov 14, 2024

yuwen01 left a comment

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 14, 2024

yuwen01 Nov 15, 2024


		Profiling the ZKVM can only be done with debug builds, and special care must be taken to ensure correctness, only one program may be profiled at a time.

		To profile a program, you have to setup a script to execute the program, many examples can be found in the repo, such as this ('fibonacci')[../../examples/fibonacci/script/src/main.rs] script.


		The `cycle-tracker` annotation is a convenient way to track cycles for specific sections of code. However, sometimes it can also be useful to track what functions are taking the most cycles across the entire program, without having to annotate every function individually.
		Profiling the VM is a good way to get an understanding of what is bottlenecking your program, and special care must be taken to ensure correctness, only one program may be profiled at a time.

Feat: Vm Profiling #1789

Are you sure you want to change the base?

Feat: Vm Profiling #1789

Conversation

nhtyy commented Nov 14, 2024

yuwen01 Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuwen01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nhtyy commented Nov 14, 2024

yuwen01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuwen01 Nov 14, 2024 •

edited

Loading