lmdb webpack issue

[bobbytreed]

Good Morning/Evening

I'm using lmdb in a vs code extension, and I'm hitting an issue where webpack doesn't package up native.js

So I see this code in the extension.js file:

let Env, Txn, Dbi, Compression, Cursor, getAddress, getBufferAddress, createBufferForAddress, clearKeptObjects, globalBuffer, setGlobalBuffer, arch, fs, os, onExit, tmpdir, lmdbError, path, EventEmitter, orderedBinary, MsgpackrEncoder, WeakLRUCache, setEnvMap, getEnvMap, getByBinary, detachBuffer, startRead, setReadCallback, write, position, iterate, prefetch, resetTxn, getCurrentValue, getCurrentShared, getStringByBinary, getSharedByBinary, getSharedBuffer, compress;
path = path__WEBPACK_IMPORTED_MODULE_0__;
let dirName = (0,path__WEBPACK_IMPORTED_MODULE_0__.dirname)((0,url__WEBPACK_IMPORTED_MODULE_1__.fileURLToPath)("file:///..path to code.../node_modules/lmdb/native.js")).replace(/dist$/, '');
let nativeAddon = node_gyp_build_optional_packages__WEBPACK_IMPORTED_MODULE_2__(dirName);

This fails after being packaged up. I know this might be a webpack issue, but I just figured you might no how to resolve this issue given your familiarity with the lmdb code which is AWESOME.

Sorry for any inconvenience.

Thank you!

[kriszyp]

Are you using webpack to run this in node or in the browser? lmdb is native binary module, written in C, so it can't run in the browser. If you are trying to run this in node, what is the failure or error?

[bobbytreed]

2023-06-14 04:35:31.419 [error] TypeError: Cannot read properties of undefined (reading 'Env')
at vu (c:\Users\robreed.vscode\extensions\skilling.learnlinter-2.8.7\out\extension.js:201255:19)
at 73241 (c:\Users\robreed.vscode\extensions\skilling.learnlinter-2.8.7\out\extension.js:201288:24)
at webpack_require (c:\Users\robreed.vscode\extensions\skilling.learnlinter-2.8.7\out\extension.js:260168:30)
at c:\Users\robreed.vscode\extensions\skilling.learnlinter-2.8.7\out\extension.js:260196:29
at Object. (c:\Users\robreed.vscode\extensions\skilling.learnlinter-2.8.7\out\extension.js:260198:3)
at u._compile (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\loader.js:4:1271)
at Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
at Module.load (node:internal/modules/cjs/loader:1058:32)
at Module._load (node:internal/modules/cjs/loader:893:12)
at f._load (node:electron/js2c/asar_bundle:2:13330)
at f._load (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:127:30006)
at E._load (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:127:26701)
at D._load (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:91:24636)
at Module.require (node:internal/modules/cjs/loader:1082:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Function.i [as __$__nodeRequire] (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\loader.js:5:98)
at n.sb (c:\Users\robreed\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:127:28020)
at async Promise.all (index 0)

let yu = Rl(
(0, n.dirname)(
(0, Dl.fileURLToPath)(
"file:///...path to module.../node_modules/lmdb/native.js"
)
).replace(/dist$/, "")
);
function vu(e) {

It's not being run in a browser, though vs code is an electron app.

The code is a .NET language server that communicates with the vs code front end.

Basically, when webpack packages up the lmdb code from lmdb, it's packaging up the native.js code.

I've had to do raw-loaders for other types of files that should just be brought over untouched, and I'm not sure this is one of those.

{
test: /.node$/,
loader: "node-loader",
},

Thank you, by the way for getting back to me!

[bobbytreed]

I added you to the repository

https://github.com/bobbytreed/test-extension

The out directory is there where you can find the native.js call out.

Please excuse the hasty coding. I was just trying to throw something together.

Thanks!

[bobbytreed]

@kriszyp Do you think you could look at this issue and provide some feedback? It looks like it might be possible to get the webpack issue resolved, but I think my knowledge of the two code bases is insufficient to resolve on my own. Maybe if you connected with Alexander

webpack/webpack#17382

[kriszyp]

It looks like the current issue is loading the name from the package.json. I am not sure why that wouldn't work in webpack, but this also seems like code that would be branching into modules for different architectures, and I don't know how that would work in webpack (are you creating distinct webpack bundles for each OS/arch?)

[bobbytreed]

I'm not, no. I'm using lmdb in a vs code extension as a super lightweight database to store auditing/inventory operations against opened repositories.

Mainly I just need to package lmdb into the .vsix file.

[kriszyp]

Are you only intending this to work on a single specific OS & architecture? Or does your bundling have a way to bundle multiple binaries?

[bobbytreed]

I don't do anything to bundle it for multiple OS architectures, but I know it is used on multiple OS architectures. VS Code publishing makes that pretty easy from what I've seen. Not much that I have to do different to have my extension work on Mac vs Windows etc.

[kriszyp]

Wouldn't bundling be trivial for platform-agnostic code (JavaScript), but different for platform-specific code (C code like LMDB)?

[bobbytreed]

I understand the logic and it makes sense. I guess that none of the other packages that I've encountered have had a requirement like this. To your point, most have been JavaScript.

[lqd1434]

It looks like the current issue is loading the name from the package.json. I am not sure why that wouldn't work in webpack, but this also seems like code that would be branching into modules for different architectures, and I don't know how that would work in webpack (are you creating distinct webpack bundles for each OS/arch?)

I get same err
"Cannot read properties of undefined (reading '0')at load.path "

so error is caused by packageName ,it is undefind @kriszyp

[kriszyp]

I have updated the build check to tolerate an undefined/missing package.json. I am not sure if that will actually address the webpack issues. I would also mention that using the download-lmdb-prebuilds bin/script might be helpful in getting all the native builds packaged in something you would distribute.

[lqd1434]

@kriszyp thanks, its work