From e14775f65dd9bbb81ec39746d9cdfd507c1801a9 Mon Sep 17 00:00:00 2001 From: Ryan Manuel Date: Mon, 1 Jul 2024 17:37:43 -0500 Subject: [PATCH] chore: add a troubleshooting guide for full v8 snapshot generation failures --- guides/v8-snapshots.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/guides/v8-snapshots.md b/guides/v8-snapshots.md index a29fc3235694..c504a503f2fc 100644 --- a/guides/v8-snapshots.md +++ b/guides/v8-snapshots.md @@ -8,13 +8,13 @@ At a high level, snapshot generation works by creating a single snapshot JS file Locally, a v8 snapshot is generated in a post install step and set up to only include node modules in the snapshot. In this way, cypress code can be modified without having to regenerate a snapshot. If you do want or need to regenerate the snapshot for development you can run: -``` +```shell yarn build-v8-snapshot-dev ``` On CI and for binary builds we run: -``` +```shell yarn build-v8-snapshot-prod ``` @@ -33,15 +33,15 @@ This cache should be maintained and updated over time. Rather than having this m ## Troubleshooting -**Local Development** +### Local Development If you're running into problems locally, either with generating the snapshot or at runtime, a good first step is to clean everything and start from scratch. This command will accomplish that (note that it will delete any new unstaged files, so if you want to keep them, either stash them or stage them): -``` +```shell git clean -fxd && yarn ``` -**Generation** +### Generation If the build v8 snapshot command is taking a long time to run on Circle CI, the snapshot cache probably needs to be updated. Run the [Update V8 Snapshot Cache](https://github.com/cypress-io/cypress/actions/workflows/update_v8_snapshot_cache.yml) github action against your branch to generate the snapshots for you on all platforms. You can choose to commit directly to your branch or alternatively issue a PR to your branch. @@ -49,7 +49,17 @@ If the build v8 snapshot command is taking a long time to run on Circle CI, the If the build v8 snapshot command fails, you can sometimes see which file is causing the problem via the stack trace. Running the build snapshot command with `DEBUG=*snap*` set will give you more information. Sometimes you can narrow the issue down to a specific file. If this is the case, you can try removing that file from the snapshot cache at `tooling/v8-snapshot/cache//snapshot-meta.json`. If this works, check in the changes and the file will get properly updated in the cache during the next automatic update. -**Runtime** +### If a Full Snapshot Rebuild Still Fails + +Occasionally, a full rebuild will still fail. This is typically a sign that there is a problem with the code we are generating, with Electron's `mksnapshot` or even deeper with the pure v8 `mksnapshot`. The first thing to be done is to figure out where the problem actually is. + +To determine if the problem is with the pure v8 `mksnapshot`, [check out](https://v8.dev/docs/source-code), [build](https://v8.dev/docs/build-gn), and run [the code](https://github.com/v8/v8/). Then you can run the built `mksnapshot` against the generated snapshot file at `tooling/v8-snapshot/cache/darwin/snapshot.js`. For example on an M1, you would run `tools/dev/gm.py arm64.release` and once that builds, you can run `mksnapshot` via: `out/arm64.release/mksnapshot `. Then, it's just a matter of figuring out where that commit is that introduces the problem. A good strategy is to check out the tag of release corresponding to the electron version where everything is working and verify it works. Then check out the tag of the release corresponding to the electron version where everything is not working and verify that it is broken. Then use a binary search on the commits in between to find the place where things break. + +To determine if the problem is with Electron's `mksnapshot`, [check out, build](https://www.electronjs.org/docs/latest/development/build-instructions-gn) and run [the code](https://github.com/electron/electron). Using [Electron's build tools](https://github.com/electron/build-tools). Is strongly recommended. When doing that an example of what the process looks like would be running `e build mksnapshot` to build the mksnapshot target and then executing `src/out/Release/mksnapshot ` to try and generate the snapshot. A good strategy is to check out the tag of release corresponding to the electron version where everything is working and verify it works. Then check out the tag of the release corresponding to the electron version where everything is not working and verify that it is broken. Then use a binary search on the commits in between to find the place where things break. If you have determined that the problem is not with the pure v8 `mksnapshot`, then the issue is likely with one of the patches that electron applied on top of v8. + +To determine if the problem is with the code we are generating, run `V8_SNAPSHOT_DISABLE_MINIFY=1 DEBUG=*snap* yarn build-v8-snapshot-prod`. The failure will spell out a command like `node /Users/user1/cypress/tooling/electron-mksnapshot/dist/mksnapshot-bin.js /Users/user1/cypress/tooling/v8-snapshot/cache/darwin/snapshot.js --output_dir /private/var/folders/9z/qzyh61x16tv5y874h_8297m40000gq/T/cy-v8-snapshot-bin`. Running that by itself will then spell out another command like `/private/var/folders/9z/qzyh61x16tv5y874h_8297m40000gq/T/mksnapshot-workdir/mksnapshot /Users/user1/cypress/tooling/v8-snapshot/cache/darwin/snapshot.js --turbo_instruction_scheduling --stress-turbo-late-spilling --target_os=mac --target_arch=arm64 --embedded_src gen/v8/embedded.S --embedded_variant Default --random-seed 314159265 --startup_blob snapshot_blob.bin --no-native-code-counters`. This last command can be used for further iterative troubleshooting. A good approach for troubleshooting involves looking at the snapshot file (`/Users/user1/cypress/tooling/v8-snapshot/cache/darwin/snapshot.js` in the above example) and commenting out various files referenced in the `// tooling/v8-snapshot/cache/darwin/snapshot-entry.js` section. If you can narrow it down to a specific file causing the problem, you can further try and narrow it down to a line of code in that file. If it is a specific line, there are two options: tweak the line in the source code to see if there's a way to cause it not to have a problem or fix the code generation at `https://github.com/cypress-io/esbuild`. + +### Runtime If you're experiencing issues during runtime, you can try and narrow down where the problem might be via a few different scenarios: