[bug] Docker release is broken starting from 9.4.0

[mmattel]

Q: when there is a new release like in the screenshot below, what is the expected update on hub.docker to make the release availabe?

Repo

hub.docker

[myoung34]

This also bit me here
It picks up releases but this project seems to consistently not publish

• 9.3.2 is released but it's not present on Docker Hub #3413
• 9.0.3 release failed #3313

[myoung34]

Seems to be related to this.

If theyre consistently failing from timeout it might be worthwhile to run the different arch docker builds simultaneously instead of in series

[robertsLando]

I'm aware of this guys and I'm sorry for the inconvenience but I don't understand what's going on, it's an issue with zui but seems something related to github actions/docker or yarn....

[myoung34]

The builds are timing out if you look at the error. Your best bet is to build them in parallel similar to how I do

it's basically an update to your yaml

[robertsLando]

What I sincerly do not understand is why they always worked and then suddenly stopped working. It happened to me on another project with the same result, with armv6/v7 install simply hangs for no reason. Notice that the test workflows (each test runs on a specific arch) end successfully so the error happens only when building all archs together with buildx. This makes me think something may have changed on action runner env

[myoung34]

It could be, but the blast radius would be smaller in parallel (they'd upload as they finish instead of failing at the very end for all if one does)

You could always try qemu/buildx instead of uraimo/run-on-arch-action@v2 and see if that helps

[robertsLando]

@myoung34 Wait, it's not run-on-arch-action that fails, it's docker build-push action

[myoung34]

This run failed to build and push for a 2 hour timeout and comes from this workflow file but youre right, i was looking at the wrong file in my previous. but you could still split them up so they all build simultaneously similar to mine

Splitting them up would also make it easier to read the logs per arch build, but aside from that there's a lot of

#33 585.0 failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
#33 585.1 ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory

which could be why it "hangs"

[robertsLando]

I already saw that and seems related to yarn, but what I don't get is firstly why it suddenly stopped working, second why that error is not always shown, it appeared only on some logs not all... sometimes build fail others it hangs, I will do more tries on Monday as I'm busy this weekend but if someone finds a solution feel free to open a PR. Sorry for the inconvenience guys

[robertsLando]

I have a feel it's something related to yarn install eating all memory on arm32: yarnpkg/berry#3972.

x64 takes few seconds to complete, arm64 takes around 5 minutes and arm32 just hangs...

[robertsLando]

Considerations

The only error I saw was: https://github.com/zwave-js/zwave-js-ui/actions/runs/6891324352/job/18756410049#step:11:773

 failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory

This only happens on armv6 and armv7 and seems related to yarn. Cool thing is I saw it only once (see the link) then no more. Could be the root cause or just something else, dunno.

Related issues:

• [Bug?]: Couldn't allocate enough memory on Windows x86 yarnpkg/berry#3972
• failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory yarnpkg/berry#4916

What I tried

• Upgrade yarn to v4 (previously was 3.2.1)
• Pre-install all deps before running buildx, remove node_modules from dockerignore and inside docker image skip the yarn install if node modules folder is present. The hang now happens on yarn rebuild command (needed for serialport)
• Tried to upgrade all gh actions to latest
• Checked if runner version changed between last succeed image and the first errored one, seems to be the same: 2.311.0

Failed: https://github.com/zwave-js/zwave-js-ui/actions/runs/6891324352/job/18756410049#step:1:1
Success: https://github.com/zwave-js/zwave-js-ui/actions/runs/6888473577/job/18737546618#step:1:1

• Tried changing docker base image, no success. Could be related to alpine maybe and may be worth tring with a non-alpine image but that will require to drop armv6 support.
• Tried to add monitoring to see if the hang is due to missing cpu/ram resources on github runner, seems not a problem: https://github.com/zwave-js/zwave-js-ui/actions/runs/6960408785#summary-18939826176
• Tried to increeese nodejs memory 386d098#diff-f34da55ca08f1a30591d8b0b3e885bcc678537b2a9a4aadea4f190806b374ddcR22

To try

• Try to run buildx locally to see if it hangs there too, if so

If anyone has any alternative I'm open to suggestions 🙌🏼

/cc @chrisns

[getlarge]

@robertsLando I don't know if they are enabled in this organization, but have you tried larger runners ?

[robertsLando]

@getlarge already checked and I don't have them, dunno if they could be enabled by @AlCalzone on organization settings?

[chrisns]

thats a pay for feature unless github are being super generous

[chrisns]

parallel building with a runner for each arch would be lovely but unless someone has an easy suggestion on how to fan out and fan back in to push a single (multi arch) docker image it doesn't really work, though would provide a full runner's compute resource to each arch build

[robertsLando]

I think a way could be to export the image as asset and import all them in the final step and push all them together.

Anyway this action is failing too and it runs on a single arch: https://github.com/zwave-js/zwave-js-ui/actions/runs/6960011576

I would be curious to know if the buildx works locally. So for now it seems the same:

amd takes a minute to complete, arm has just complete now while I'm writing (took 6x the time of amd). Lets see if the others keep for more then an hour

[robertsLando]

Ok nice finding... using npm rebuild in place of yarn rebuild has reduced build time for arm64 from 360s to 150s. Let's see how it goes for arm32...

UPDATE: still not working

[robertsLando]

Fixed by #3443 🎉

[snowmangh]

Hello. Looks like this issue has been resolved? I just tried and getting the same error 400.

[chrisns]

400 on trying to pull from docker hub?
What Arch?
I'd generally suggest using ghcr instead of docker to avoid rate caps etc

[snowmangh]

@chrisns Yes, from docker.io zwavejs/zwave-js-ui:latest. I always used this method without issues, so far via Portainer.

What Arch? Apollolake

[robertsLando]

@snowmangh run uname -a to find the arch. Anyway it is on docker so you may be having some other issue

[snowmangh]

@robertsLando Here's what I got.

Linux Defiant 4.4.302+ #69057 SMP Mon Nov 13 14:21:00 CST 2023 x86_64 GNU/Linux synology_apollolake_918+

BTW, I just added ghcr.io in Portainer/Registries and pulled this image "zwave-js/zwave-js-ui:9.5.1". It worked no problem.