polkavm-test-data #240

jarkkojs · 2024-12-15T14:33:21Z

Closes: #209

jarkkojs · 2024-12-15T14:43:04Z

I don't know how to fix build-and-test-windows.

athei · 2024-12-16T12:02:28Z

The windows runner doesn't seem to use a rustup managed cargo. So they are not able to download and use the toolchain specified in the rustup-toolchain.toml. So you might have to make sure that rustup is used on windows.

koute · 2025-01-01T07:19:32Z

crates/polkavm-test-data/build.rs

+    let mut cmd = Command::new("cargo");
+
+    cmd.current_dir(project_path)
+        .env_clear()


Why the env_clear?

Not sure. I need refresh on Monday. It's a strong possibility that it is for no good reason ;-)

The history for this is that I derived the code from substrate/frame/revive/fixtures/build.rs.

I guess the argument here (and fixtures) is that by doing this we remove any side-effects of environment variables set by parent i.e. always starting from the same state.

I recall @athei did the code I used as basis so maybe he has a comment for this...

Yeah it makes sense since we don't want other variables meant for other targets to influence this build.

See my remarks below.

crates/polkavm-test-data/build.rs

koute · 2025-01-01T07:42:02Z

I don't know how to fix build-and-test-windows.

This is weird; it's clearly using rustup because cargo test --all triggers a toolchain install:

>> cargo test (debug)
info: syncing channel updates for '1.75.0-x86_64-pc-windows-msvc'
info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)

but when it's ran from inside of the build.rs it ignores the guest-programs/rust-toolchain.toml for some reason?

I'm not sure how to fix it; you'll have to fiddle with it. Maybe clearing the environment variables screws it up?

athei · 2025-01-02T19:29:25Z

This is weird; it's clearly using rustup because cargo test --all triggers a toolchain install:

Oops didn't see that and jumped to conclusions. Rustup seems to installed and is picking up the toolchain file in the repo root.

but when it's ran from inside of the build.rs it ignores the guest-programs/rust-toolchain.toml for some reason?

Yes it is weird. Maybe some platform differences regarding pathes?

I'm not sure how to fix it; you'll have to fiddle with it. Maybe clearing the environment variables screws it up?

We do the clearing in the other build scripts in polkadot-sdk, too. But I don't think it is ever tested on Windows. Yeah maybe removing the clearing and work from there. Maybe additional or different variables need to be forwarded on Windows.

crates/polkavm-test-data/build.rs

athei · 2025-01-07T08:17:21Z

At least now it is failing everywhere? :D

jarkkojs · 2025-01-07T11:05:06Z

At least now it is failing everywhere? :D

Working on it... temporary glitch ;---)

jarkkojs · 2025-01-08T17:03:48Z

@koute, @athei, I'll go a few diff's now that I've fixed the stuff I know how to fix!

Diff 1:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..4971115 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -21,7 +21,6 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
     let mut cmd = Command::new("cargo");
 
     cmd.current_dir(project_path)
-        .env_clear()
         .env("PATH", std::env!("PATH"))
         .env("RUSTFLAGS", rust_flags)
         .env("RUSTUP_HOME", std::env!("RUSTUP_HOME"))

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data*
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

Diff 2:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..b637903 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -18,10 +18,9 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
 
     // Note that using `std::env!("CARGO")` fails the compilation since the
     // compilation requires nightly.
-    let mut cmd = Command::new("cargo");
+    let mut cmd = Command::new(std::env!("CARGO"));
 
     cmd.current_dir(project_path)
-        .env_clear()
         .env("PATH", std::env!("PATH"))
         .env("RUSTFLAGS", rust_flags)
         .env("RUSTUP_HOME", std::env!("RUSTUP_HOME"))

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data* 24s
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

Diff 3:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..a901351 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -18,7 +18,7 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
 
     // Note that using `std::env!("CARGO")` fails the compilation since the
     // compilation requires nightly.
-    let mut cmd = Command::new("cargo");
+    let mut cmd = Command::new(std::env!("CARGO"));
 
     cmd.current_dir(project_path)
         .env_clear()

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data* 19s
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

How should I move forward?

athei · 2025-01-09T12:58:34Z

How should I move forward?

Debug this further. Probably reproducing in a Windows VM. Looking at the code nothing obvious comes to mind for me.

jarkkojs · 2025-01-09T16:06:46Z

How should I move forward?

Debug this further. Probably reproducing in a Windows VM. Looking at the code nothing obvious comes to mind for me.

I'm working on setting up a Windows VM. It is worth of investment as Windows could break also in future because POSIX does not always align that well in that environment.

So yeah, we're on the same page.

jarkkojs · 2025-01-11T00:16:45Z

I spent a lot of time on setting up a Windows environment that is usable. Now I have an image that logins automatically and boots to the terminal, and I can reach the host via ssh. Also environment has been setup.

This what happens:

@athei, @koute: So I guess this maps to replacing std::env!("HOME") with std::env::home_dir() doesn't it?

jarkkojs · 2025-01-11T00:17:40Z

I think it was useful spend couple of days to figure out effective way to use for Windows. After this I can reproduce bugs in no time. It was an investment ;-)

jarkkojs · 2025-01-18T17:56:44Z

Without this Linux build will act the same (will complain about -Z flags):

    if !cfg!(target_os = "windows") {
        cmd.env_clear()
            .env("PATH", std::env::var("PATH").unwrap())
            .env("RUSTUP_HOME", std::env::var("RUSTUP_HOME").unwrap());
    }

So better to have this initial environment setup branched and find the right recipe for Windows.

jarkkojs · 2025-01-18T18:25:22Z

@athei, @koute, possible success I got into brand new error:

error: failed to run custom build command for `polkavm-test-data v0.19.0 (D:\a\polkavm\polkavm\crates\polkavm-test-data)`

Caused by:
  process didn't exit successfully: `D:\a\polkavm\polkavm\target\debug\build\polkavm-test-data-38f55663277395bd\build-script-build` (exit code: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR
  cargo::error=info: syncing channel updates for 'nightly-2024-11-01-x86_64-pc-windows-msvc'
  error: could not download file from 'https://static.rust-lang.org/dist/2024-11-01/channel-rust-nightly.toml.sha256' to 'C:\Users\runneradmin\.rustup\tmp\p0cb_8xh58tanwbo_file': failed to make network request: error sending request for url (https://static.rust-lang.org/dist/2024-11-01/channel-rust-nightly.toml.sha256): client error (Connect): dns error: A non-recoverable error occurred during a database lookup. (os error 11003): A non-recoverable error occurred during a database lookup. (os error 11003)

warning: build failed, waiting for other jobs to finish...
Error: Process completed with exit code 101.

I removed RUSTUP_HOME setting from initial environment setup to get here.

It is very promising given that it is now going after nightly and not complaining about -Z.

Signed-off-by: Jarkko Sakkinen <[email protected]>

jarkkojs · 2025-01-19T03:55:24Z

Better to take a break with this one, nothing comes to mind ATM to try out with Windows build. Revisiting when new idea pop up.

jarkkojs requested review from koute and athei December 15, 2024 14:42

jarkkojs requested a review from aman4150 December 15, 2024 14:45

jarkkojs force-pushed the polkavm-test-data branch 4 times, most recently from c5fe193 to df084cf Compare December 16, 2024 10:12

jarkkojs mentioned this pull request Dec 20, 2024

Fix debug info support #208

Open

koute reviewed Jan 1, 2025

View reviewed changes

athei reviewed Jan 2, 2025

View reviewed changes

crates/polkavm-test-data/build.rs Show resolved Hide resolved

jarkkojs force-pushed the polkavm-test-data branch 2 times, most recently from 32a763d to f7481c7 Compare January 6, 2025 23:16

jarkkojs force-pushed the polkavm-test-data branch 5 times, most recently from e5b98a7 to c8f485b Compare January 8, 2025 16:57

jarkkojs requested review from athei and koute January 8, 2025 17:04

jarkkojs force-pushed the polkavm-test-data branch 14 times, most recently from 3951732 to 2c921da Compare January 18, 2025 17:55

jarkkojs force-pushed the polkavm-test-data branch 5 times, most recently from 1a7d5c0 to 85fe529 Compare January 18, 2025 18:19

jarkkojs force-pushed the polkavm-test-data branch 6 times, most recently from bc61f52 to 5022f31 Compare January 18, 2025 20:25

polkavm-test-data

302c305

Signed-off-by: Jarkko Sakkinen <[email protected]>

jarkkojs force-pushed the polkavm-test-data branch from 5022f31 to 302c305 Compare January 18, 2025 20:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

polkavm-test-data #240

polkavm-test-data #240

jarkkojs commented Dec 15, 2024 •

edited

Loading

jarkkojs commented Dec 15, 2024

athei commented Dec 16, 2024

koute Jan 1, 2025

jarkkojs Jan 4, 2025

jarkkojs Jan 6, 2025

jarkkojs Jan 6, 2025

athei Jan 7, 2025

jarkkojs Jan 8, 2025

koute commented Jan 1, 2025 •

edited

Loading

athei commented Jan 2, 2025

athei commented Jan 7, 2025

jarkkojs commented Jan 7, 2025

jarkkojs commented Jan 8, 2025 •

edited

Loading

athei commented Jan 9, 2025 •

edited

Loading

jarkkojs commented Jan 9, 2025 •

edited

Loading

jarkkojs commented Jan 11, 2025 •

edited

Loading

jarkkojs commented Jan 11, 2025

jarkkojs commented Jan 18, 2025

jarkkojs commented Jan 18, 2025 •

edited

Loading

jarkkojs commented Jan 19, 2025

polkavm-test-data #240

Are you sure you want to change the base?

polkavm-test-data #240

Conversation

jarkkojs commented Dec 15, 2024 • edited Loading

jarkkojs commented Dec 15, 2024

athei commented Dec 16, 2024

koute Jan 1, 2025

Choose a reason for hiding this comment

jarkkojs Jan 4, 2025

Choose a reason for hiding this comment

jarkkojs Jan 6, 2025

Choose a reason for hiding this comment

jarkkojs Jan 6, 2025

Choose a reason for hiding this comment

athei Jan 7, 2025

Choose a reason for hiding this comment

jarkkojs Jan 8, 2025

Choose a reason for hiding this comment

koute commented Jan 1, 2025 • edited Loading

athei commented Jan 2, 2025

athei commented Jan 7, 2025

jarkkojs commented Jan 7, 2025

jarkkojs commented Jan 8, 2025 • edited Loading

athei commented Jan 9, 2025 • edited Loading

jarkkojs commented Jan 9, 2025 • edited Loading

jarkkojs commented Jan 11, 2025 • edited Loading

jarkkojs commented Jan 11, 2025

jarkkojs commented Jan 18, 2025

jarkkojs commented Jan 18, 2025 • edited Loading

jarkkojs commented Jan 19, 2025

jarkkojs commented Dec 15, 2024 •

edited

Loading

koute commented Jan 1, 2025 •

edited

Loading

jarkkojs commented Jan 8, 2025 •

edited

Loading

athei commented Jan 9, 2025 •

edited

Loading

jarkkojs commented Jan 9, 2025 •

edited

Loading

jarkkojs commented Jan 11, 2025 •

edited

Loading

jarkkojs commented Jan 18, 2025 •

edited

Loading