BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04 #990

rathboma · 2023-04-11T15:49:11Z

From 8.3.0 onward the build image is now Ubuntu 20.04 as GitHub Actions has removed Ubuntu 18.04 support.

Unfortunately this brings back an old set of issues -- any app using the prebuilds of better-sqlite3 will experience crashes when distributed through Snap, or when running on a system that contains a lower version of glibc (which is a LOT of systems).

This will specifically be a problem for those running prebuilds on servers with Ubuntu 16.04, 18.04, 14.04, or Debian Buster, all of which are still supported and used in many businesses. It is also a problem for Electron apps (mine included)

Context: Glibc 2.29 is a breaking change, but is only present on newer OSes.

SQLite3 is perfectly happy being built with older versions of glibc, so maybe there's a way to get those older versions on the build machines, or to build inside a docker container using a Ubuntu-18.04 or Debian buster image?

Related, & previously resolved by downgrading to Ubuntu 18.04 build images:

mceachen · 2023-04-11T17:05:42Z

Thanks for taking the time to do this research--apologies for the regression.

~~Can you submit a PR that reverts #982 ? @JoshuaWise has granted me a small subset of permissions for this repo--I can't self-approve any changes, for example.~~

rathboma · 2023-04-11T17:53:37Z

@mceachen I don't think it can be reverted. GitHub no longer supports the 18.04 build image

mceachen · 2023-04-11T18:25:33Z

SADNESS actions/runner-images#6002

rathboma · 2023-04-11T18:29:02Z

Yeah, that's why I wanted to start the discussion again, because I'm not sure this is going to be super easy to solve without some bigger build changes :-/

mceachen · 2023-04-11T18:34:07Z

I was hoping that glibc had some sort of autoconf/environment variable magicks for GLIBC_V2.21_MODE, but it seems that newer glibc removed older kernel support, so that's not going to happen.

All my googling seems to point to using a VM/docker instance to restore ubuntu 18 builds.

rathboma · 2023-04-11T18:35:39Z

Yeah same. Seems like it is either that or a CHROOT with a specific version of glibc installed.

I think 2.29 introduced a bunch of new stuff that isn't backwards compatible, because usually they're very good about backwards compatibility

rathboma · 2023-04-11T18:37:11Z

Or I guess installing a local version of an older glibc somewhere and using that one?

mceachen · 2023-04-11T22:01:35Z

I looked at this a bit more.

I would suggest we drop Ubuntu 18 support. EOL is at the end of this month. This will be @JoshuaWise 's call.

If we do want to try to kick the can down the road a couple months, we can push this out until september by adding this GHA job:

  prebuild-glibc:
    name: Prebuild on ancient glibc
    runs-on: ubuntu-latest
    container: node:16-stretch
    needs: publish
    steps:
      - uses: actions/checkout@v3
      - run: apt-get update && apt-get install build-essential git python3
      - run: npm install --ignore-scripts
      - run: ${{ NODE_BUILD_CMD }} -u ${{ secrets.GITHUB_TOKEN }}

The stretch build has 2.24:

$ docker run -it node:16-stretch sh
# head -1 /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
# ldd --version
ldd (Debian GLIBC 2.24-11+deb9u4) 2.24

rathboma · 2023-04-11T22:41:22Z

Great! I think pushing it out is a great idea.

Standard support for 18.04 ends soon, but ESM support runs for a few more years, many individuals get this for free, and most enterprises use it also. 14.04 is still supported with ESM until 2024!

rathboma · 2023-04-11T22:41:41Z

Never seen that container directive before. I'm going to try that myself

mceachen · 2023-04-11T22:43:35Z

I didn't test the new draft job yet, but see c99fca6 for what I was thinking.

rathboma · 2023-04-11T23:17:50Z

Looks great. It's very frustrating to have to micro manage builds like this right? I have the same frustrations, lol

m4heshd · 2023-04-12T22:12:57Z

@mceachen This is why I explicitly mentioned this HUGE breaking change in the PR note. (#982)

This change is necessary since GitHub has dropped support for the ubuntu-18.04 image. (actions/runner-images#6002)

This will also mean that Linux distributions with only glibc version 2.29 and above will be supported by the prebuilt binaries.

I've shopped around for many workarounds for this before the change. Unfortunately, we have to go along with all these deprecations and there's no way around it. I've even looked at using a custom runner image (I usually waste a ton of time looking for workarounds even when they're quite unnecessary). But that's practically useless since Node 18 itself requires glibc >= 2.28 and Node 16 is soon EOL. That timeline unfortunately didn't line up with GH's support drop for the Ubuntu image. Polluting the build workflow for mere short-term backward compatibility just felt wrong. But that's just my way of doing things. I would love some input from @JoshuaWise on this.

@rathboma If this is a necessity for your application, I think a solution on the application's scope would fit much better since you have full control of the building, packaging, and distribution of your app. This is just my own opinion. I never use prebuilt binaries on my distributions. I always rebuild them while packaging. You can use a self-hosted runner while doing that to please your requirements. Please correct me if I'm missing anything here because I have no experience in distributing in packmans like Snap.

lyswhut · 2023-05-06T06:55:30Z

@mceachen Hi, i tried this configuration:

https://github.com/mceachen/better-sqlite3/blob/c99fca6b901201505537da1628c8b8e6f688faf7/.github/workflows/build.yml#L86-L96

But it failed to build because the highest Python version of Debian 9's official package manager source is v3.5.x, and node-gyp v8 needs Python 3.6 or higher to work. Here are some reference links

rathboma · 2023-05-09T15:30:10Z

@m4heshd Yeah I'm going to have to build the package myself in my runner. I've had a hard time in the past telling my build tools to NOT use prepacked binaries, I don't think the tooling does this very well.

For now I'm just going to lock to a specific version. I don't work on my app full time, so this type of stuff is a net negative on productivity :-(.

Is there a way to mark prebuilt binaries as not working for certain operating systems?

All the better-sqlite3 contributors - thanks for doing such an amazing job. I realize this stuff isn't your fault. :-)

JounQin · 2023-05-10T12:17:50Z

Can we republish v8.2.0 as v8.3.1 and then release current v8.3.0 as v9.0.0 for its breaking change?

pixdrift · 2023-08-18T02:41:51Z

I appreciate that this thread is Ubuntu/Debian focused, but I wanted to include a note that this change also breaks install for EL8 variants as they ship with GLIBC 2.28.

This includes Red Hat Enterprise Linux, CentOS Stream, Oracle Linux, Alma, Rocky etc.

Red Hat Enterprise Linux 8 has full support from Red Hat until May 2024 and at that date it goes into extended support (but is still supported). Not that vendor support governs decisions of support of these builds, but wanted to mention it.
https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle

I have been able to replicate the install issue in RHEL 8.x and that is what brought me to this ticket.

Something that may also be worth mentioning is the assumptions made in the fallback behaviour when an incompatible GLIBC is detected. This led to a cascade of issues as it attempted to build as it expects python (mentioned above), make, gcc, and open Internet access to pull sources.

flaviut · 2023-09-25T21:37:28Z

I also want to note that the AWS Lambda node 18.x runtime runs on Amazon Linux 2, which only has glibc 2.26.

This is outside of node18's support (glibc 2.28 is required), and I definitely don't expect this project to take any action based on that, but I wanted to bring it up as an example of another distro with old glibc versions.

anonghuser · 2023-09-30T21:47:50Z

Getting this error on Oracle Linux 8 which has glibc 2.28

~~Aside of waiting for you to fix the prebuilt binaries, can you describe how people should configure it to be built locally with their own version of glibc?~~ The preinstall script from docs/compilation.md seems incorrect for people that prefer pnpm or other package managers.

[EDIT: Apologies, I only saw the first error in the error log and did not notice a further error about missing g++. I see now that if the prebuild binaries are not usable, you attempt a local build and it was just failing. Installing gcc-c++ worked to fix it.]

anonghuser · 2023-09-30T22:20:02Z

It seems to me even though github may have dropped support for old ubuntu versions as the base of its github actions runners, it still supports basically any docker image as an additional service:
https://docs.github.com/en/actions/using-containerized-services/about-service-containers
https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container
So you should still be able to build on older ubuntu docker images.

And just for completeness, though I'm sure your google-fu is no worse than mine and you've probably already seen this, there are several other alternatives described here:
https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version
The more interesting-sounding ones are actually not the top answers but further down, i.e. using zig c++ or glibc_version_header. The most practical solution still seems building in docker though.

viceice · 2024-07-02T10:21:13Z

Is it expected that better-sqlite v11 needs GLIBC v2.33+

#20 [build-slim build 3/8] RUN corepack pnpm fetch --prod
       #20 116.7 Progress: resolved 776, reused 0, downloaded 320, added 316
       #20 118.5 Progress: resolved 776, reused 0, downloaded 342, added 324
       #20 119.8 Progress: resolved 776, reused 0, downloaded 354, added 351
       #20 121.3 Progress: resolved 776, reused 0, downloaded 407, added 383
       #20 123.7 Progress: resolved 776, reused 0, downloaded 414, added 406
       #20 125.0 Progress: resolved 776, reused 0, downloaded 449, added 444
       #20 126.4 Progress: resolved 776, reused 0, downloaded 485, added 477
       #20 127.8 Progress: resolved 776, reused 0, downloaded 505, added 488
       #20 130.0 Progress: resolved 776, reused 0, downloaded 517, added 517
       #20 131.3 Progress: resolved 776, reused 0, downloaded 563, added 546
       #20 133.4 Progress: resolved 776, reused 0, downloaded 592, added 576
       #20 134.4 Progress: resolved 776, reused 0, downloaded 609, added 608
       #20 135.9 Progress: resolved 776, reused 0, downloaded 614, added 609
       #20 137.5 Progress: resolved 776, reused 0, downloaded 640, added 615
       #20 139.9 Progress: resolved 776, reused 0, downloaded 671, added 646
       #20 141.0 Progress: resolved 776, reused 0, downloaded 677, added 676
       #20 142.2 Progress: resolved 776, reused 0, downloaded 678, added 676
       #20 144.0 Progress: resolved 776, reused 0, downloaded 724, added 720
       #20 146.4 Progress: resolved 776, reused 0, downloaded 736, added 736
       #20 148.0 Progress: resolved 776, reused 0, downloaded 752, added 752
       #20 149.0 Progress: resolved 776, reused 0, downloaded 775, added 775
       #20 150.0 Progress: resolved 776, reused 0, downloaded 776, added 775
       #20 151.3 Progress: resolved 776, reused 0, downloaded 776, added 776
       #20 151.3 Progress: resolved 776, reused 0, downloaded 776, added 776, done
       #20 155.4 .../node_modules/protobufjs postinstall$ node scripts/postinstall
       #20 155.4 .../node_modules/core-js-pure postinstall$ node -e "try{require('./postinstall')}catch(e){}"
       #20 155.9 .../node_modules/better-sqlite3 install$ prebuild-install || node-gyp rebuild --release
       #20 155.9 .../.pnpm/[email protected]/node_modules/re2 install$ install-from-cache --artifact build/Release/re2.node --host-var RE2_DOWNLOAD_MIRROR --skip-path-var RE2_DOWNLOAD_SKIP_PATH --skip-ver-var RE2_DOWNLOAD_SKIP_VER || node-gyp -j max rebuild
       #20 158.0 .../node_modules/protobufjs postinstall: Done
       #20 158.1 .../node_modules/core-js-pure postinstall: Done
       #20 159.2 .../.pnpm/[email protected]/node_modules/re2 install: Trying https://github.com/uhop/node-re2/releases/download/1.21.3/linux-arm64-108.br ...
       #20 161.5 .../node_modules/better-sqlite3 install: prebuild-install info begin Prebuild-install version 7.1.2
       #20 161.6 .../node_modules/better-sqlite3 install: prebuild-install info looking for local prebuild @ prebuilds/better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 161.7 .../node_modules/better-sqlite3 install: prebuild-install info npm cache directory missing, creating it...
       #20 161.7 .../node_modules/better-sqlite3 install: prebuild-install info looking for cached prebuild @ /root/.npm/_prebuilds/1c8951-better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 161.7 .../node_modules/better-sqlite3 install: prebuild-install http request GET https://github.com/WiseLibs/better-sqlite3/releases/download/v11.0.0/better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 162.9 .../.pnpm/[email protected]/node_modules/re2 install: Writing to build/Release/re2.node ...
       #20 164.0 .../node_modules/better-sqlite3 install: prebuild-install http 200 https://github.com/WiseLibs/better-sqlite3/releases/download/v11.0.0/better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 164.2 .../node_modules/better-sqlite3 install: prebuild-install info downloading to @ /root/.npm/_prebuilds/1c8951-better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz.120-597ea11370291.tmp
       #20 164.6 .../node_modules/better-sqlite3 install: prebuild-install info renaming to @ /root/.npm/_prebuilds/1c8951-better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 164.6 .../node_modules/better-sqlite3 install: prebuild-install info unpacking @ /root/.npm/_prebuilds/1c8951-better-sqlite3-v11.0.0-node-v108-linux-arm64.tar.gz
       #20 165.4 .../node_modules/better-sqlite3 install: prebuild-install info unpack resolved to /usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/better-sqlite3/build/Release/better_sqlite3.node
       #20 165.4 .../node_modules/better-sqlite3 install: prebuild-install warn install /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/better-sqlite3/build/Release/better_sqlite3.node)
       #20 168.3 .../node_modules/better-sqlite3 install: gyp info it worked if it ends with ok
       #20 168.3 .../node_modules/better-sqlite3 install: gyp info using [email protected]
       #20 168.3 .../node_modules/better-sqlite3 install: gyp info using [email protected] | linux | arm64
       #20 170.9 .../node_modules/better-sqlite3 install: gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"

viceice · 2024-07-02T10:33:31Z

docker run --rm -it node:18 bash                                                                                                     ─╯
root@dfbdcde88d4d:/# ldd  --version
ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
root@dfbdcde88d4d:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@dfbdcde88d4d:/#
exit

i think you should use node:18-bullseye as base image to have GLIBC v2.31

docker run --rm -it node:18-bullseye bash                                                                                            ─╯
Unable to find image 'node:18-bullseye' locally
18-bullseye: Pulling from library/node
8c5542ca7520: Download complete
414da18c1936: Download complete
8a305f523084: Download complete
c1c1a7d83fb1: Download complete
6a8ddfe2c2d3: Download complete
fc779bf5bf49: Download complete
da238dd9d1f5: Download complete
1e5a1517e423: Download complete
Digest: sha256:6f8868da0475253e114d94c237c9048034adb27265484274874380078bdd40d7
Status: Downloaded newer image for node:18-bullseye
root@78ab76cfd442:/# ldd --version
ldd (Debian GLIBC 2.31-13+deb11u10) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
root@78ab76cfd442:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

better-sqlite3/.github/workflows/build.yml

Line 164 in 309cb37

    
                     docker run --rm -v $(pwd):/tmp/project --entrypoint /bin/sh --platform linux/${{ matrix.arch }} node:18 -c "\

mceachen added a commit to mceachen/better-sqlite3 that referenced this issue Apr 11, 2023

draft for WiseLibs#990

c99fca6

h3poteto mentioned this issue May 1, 2023

Failed to launch on Ubuntu h3poteto/whalebird-desktop#4338

Closed

lyswhut mentioned this issue May 12, 2023

[Bug]: linux x64版本无法打开软件 lyswhut/lx-music-desktop#1365

Open

2 tasks

NathanaelA mentioned this issue Jul 15, 2023

Using better-sqlite3 in a global CLI causes errors. #1015

Open

trinhanhngoc mentioned this issue Oct 5, 2023

version `GLIBC_2.29' not found odoo-ide/vscode-odoo#34

Closed

dziudek mentioned this issue Nov 12, 2023

[Bug]: JS error GLIBC on ubuntu 20.04 GetPublii/Publii#1294

Closed

andelf mentioned this issue Jan 15, 2024

Getting warning "Search index error, please manually delete" logseq/logseq#10851

Open

2 tasks

viceice mentioned this issue Jul 2, 2024

Use node bullseye to build arm images #1216

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04 #990

BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04 #990

rathboma commented Apr 11, 2023 •

edited

Loading

mceachen commented Apr 11, 2023 •

edited

Loading

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023 •

edited

Loading

rathboma commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

m4heshd commented Apr 12, 2023

lyswhut commented May 6, 2023

rathboma commented May 9, 2023

JounQin commented May 10, 2023

pixdrift commented Aug 18, 2023

flaviut commented Sep 25, 2023

anonghuser commented Sep 30, 2023 •

edited

Loading

anonghuser commented Sep 30, 2023 •

edited

Loading

viceice commented Jul 2, 2024

viceice commented Jul 2, 2024 •

edited

Loading

BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04 #990

BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04 #990

Comments

rathboma commented Apr 11, 2023 • edited Loading

mceachen commented Apr 11, 2023 • edited Loading

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023 • edited Loading

rathboma commented Apr 11, 2023

rathboma commented Apr 11, 2023

mceachen commented Apr 11, 2023

rathboma commented Apr 11, 2023

m4heshd commented Apr 12, 2023

lyswhut commented May 6, 2023

rathboma commented May 9, 2023

JounQin commented May 10, 2023

pixdrift commented Aug 18, 2023

flaviut commented Sep 25, 2023

anonghuser commented Sep 30, 2023 • edited Loading

anonghuser commented Sep 30, 2023 • edited Loading

viceice commented Jul 2, 2024

viceice commented Jul 2, 2024 • edited Loading

rathboma commented Apr 11, 2023 •

edited

Loading

mceachen commented Apr 11, 2023 •

edited

Loading

mceachen commented Apr 11, 2023 •

edited

Loading

anonghuser commented Sep 30, 2023 •

edited

Loading

anonghuser commented Sep 30, 2023 •

edited

Loading

viceice commented Jul 2, 2024 •

edited

Loading