Nightly builds started failing yesterday - manual build also fails

[Jaifroid]

For last two nights, the nightly run has been failing at the same spot as per screenshot below.

Running the build nightly makefile script manually gives a bit more info about a specific (and somewhat cryptic for me) assertion that fails:

wasm2js: /b/s/w/ir/cache/builder/emscripten-releases/binaryen/src/ir/block-utils.h:54: wasm::Expression *wasm::BlockUtils::simplifyToContents(wasm::Block *, T *, bool) [T = wasm::Vacuum]: Assertion `block->type.isConcrete() && singleton->type == Type::unreachable' failed.
em++: error: '/emsdk/upstream/bin/wasm2js --emscripten -O libzim-asm.wasm --mvp-features --enable-threads --enable-mutable-globals --enable-bulk-memory --enable-sign-ext' failed (received SIGABRT (-6))
make: *** [Makefile:95: libzim-asm.js] Error 1

@mgautierfr Are you aware of anything having changed upstream with the build of libzim_wasm_emscripten on or after 18th June that could help pinpoint the issue here?

[mgautierfr]

Are you aware of anything having changed upstream with the build of libzim_wasm_emscripten on or after 18th June that could help pinpoint the issue here?

No. But wasm::BlockUtils::simplifyToContents seems to be internal to wasm. Maybe a broken update of wasm ?

[Jaifroid]

Hmm. Though it's only the nightly that has started failing. I ran a release build using libzim_wasm-emscripten-8.2.0-1.tar.gz, and that builds just fine: https://github.com/openzim/javascript-libzim/actions/runs/5319878042/jobs/9632904538.

Also, we use a pinned version of the EMSDK - emscripten/emsdk:3.1.12, which shouldn't have changed.

I'll see if we can build against the last working nightly which was libzim_wasm-emscripten-2023-06-18.tar.gz and report back.

[mgautierfr]

Last commit on libzim is openzim/libzim@55f098e and is two days ago. There is nothing special in it.

[mgautierfr]

May be related to emscripten-core/emscripten#15768 (but I haven't read all the thread yet)

[Jaifroid]

OK, so when I build a nightly build with libzim_wasm-emscripten-2023-06-18.tar.gz, it all builds successfully:

And then when I run the same set of commands using libzim_wasm-emscripten-2023-06-20.tar.gz, it fails:

What could be the difference? Might some upstream library used to build the libzim wasm binary have changed?

[mgautierfr]

Can you remove the -s WASM=0 ? It seems to cause of emscripten-core/emscripten#15768 (among other things)

[Jaifroid]

Ah, yes, building the WASM version is still possible with latest nightly, whereas building the ASM version is no longer possible (see screenshot below). It is indeed the ASM versions that started failing two days ago.

This doesn't explain what changed a couple of days ago that disrupts the building of ASM (and it is definitely a change upstream, something in the binary). Building an ASM version would be important if we are ever to support older browsers. This would require two things: a) build a non-p-threaded version #43, and b) build this ASM version.

I could disable ASM for now, but it would be better if we could understand what the upstream cause of this is!

[mgautierfr]

You should probably open a issue on emscripten upstream (or update emscripten-core/emscripten#15768)

[Jaifroid]

I can't really open an issue if I don't know what is causing the error upstream. Code in this Repo compiles the ASM just fine if I use the Released version or nightly of two nights ago. It's the latest nightlies that don't compile to ASM only... But there has been no significant change upstream. Nevertheless, completely reproducible with latest nightlies only. Very strange! I can try using a later version of Emscripten, but #9 could still be a blocker there in any case. Needs testing.

[kelson42]

@mgautierfr @Jaifroid Do we use a moving version of emscripten?! Our dependences should be fixed on a very specific version.

[Jaifroid]

@kelson42, no we use a pinned version 3.1.12: https://github.com/openzim/javascript-libzim/blob/main/docker/Dockerfile#L2

[kelson42]

So, the trigger is by us.

[Jaifroid]

Some good news: updating the Emscripten SDK used in this Repo to latest (3.1.41) allows us to build both release and nightly versions! Today's nightly 22-06-23 is building OK with this: https://github.com/openzim/javascript-libzim/actions/runs/5348242381/jobs/9697864250 . I just need to test the built artefacts that have been published by this run in https://download.openzim.org/nightly/2023-06-22/ just now and will report back shortly.

[kelson42]

@Jaifroid So we can close #9?

[Jaifroid]

Confirmed that both the release versions (built from latest libzim release) and the nightly versions (build from libzim nightly 22-06-23) work fine (tested with the prototype provided in this Repo).

So I can speculate that the version of Emscripten used to build the libzim.a binary in kiwix build is out of sync with the pinned version we were using here.

@kelson42 The PR #52, when complete, will close both this issue and #9 when merged. There is still some work to be done: I am getting some errors when trying to build DEV versions (unminified) of javascript-libzim due to flags that are no longer supported on the latest Emscripten. But I know these are not blockers because we can build production (minified) versions of both release and nightly without error (though with several warnings, some of which may need attention).

[Jaifroid]

Awaiting nightly to be sure it's fixed. The fix breaks building from source currently (#9), but that is the lesser of two evils.

[Jaifroid]

Nightly build worked.