fix: electron-builder fails when list of node_modules files is too big to pass in a glob

[beyondkmp]

fix #8705

root cause

When listing all files, if there are too many files, it will trigger glob's length limit.

how to fix

If all files in a folder are unpacked, use the folder name directly instead of listing all files.

[changeset-bot]

🦋 Changeset detected

Latest commit: 7be5628

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
app-builder-lib	Patch
dmg-builder	Patch
electron-builder-squirrel-windows	Patch
electron-builder	Patch
electron-forge-maker-appimage	Patch
electron-forge-maker-nsis-web	Patch
electron-forge-maker-nsis	Patch
electron-forge-maker-snap	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[beyondkmp]

The main issue in this MR is caused by users directly configuring the entire node_modules in the asarUnpack pattern.

asarUnpack supports '!', but electron/asar doesn't support this '!'. This problem will occur whenever users have directories containing a large number of files.

My thought is whether electron-builder's asarUnpack should also not support '!' patterns, keeping consistent with electron/asar. @mmaietta Do you have any good ideas or suggestions?

[mmaietta]

Is there a way we can resolve ! filters before passing to electron/asar's minimatch?

It seems like a pretty hefty decrease in electron-builder functionality if we remove the ! filter feature?

[mmaietta]

Can you clarify this area of code with a comment? It looks like we're determining whether all files in the fileSet are being unpacked, if so, then we glob them together with an umbrella glob destination/** to reduce the number of unpacked paths that are being passed to minimatch?

[beyondkmp]

Yes, exactly as you mentioned.

electron/asar doesn't implement ! filtering. Having to do the conversion at the electron-builder layer is very complicated, and there's currently no good method for conversion.

Therefore, we can only solve this problem from a different angle. Exceeding the length limit is always caused by filtering one or more directories. If we're just filtering specific files, it won't exceed the length limit. So solving these directory filtering issues would solve this problem.

Currently, there might still be some edge cases, such as when a directory contains tens of thousands of files, and the requirement is to put just a few files from this directory into asar while putting all other files into asar.unpack.
In practical use, this situation probably won't occur - it's rare to have a directory with tens of thousands of files where only a few files need to be in asar.

[mmaietta]

For unpackedPaths.add(${destination}${path.sep}**), is it possible to use ${relative}${path.sep}** to shorten the minimatch paths even further? I can't recall off the top of my head whether it requires absolute paths or if relative works.

[beyondkmp]

Good idea. However, we need to use **${path.sep}${relative}${path.sep}**.

I've changed files to use this format as well, and current testing shows no issues.

electron-builder/packages/app-builder-lib/src/asar/asarUtil.ts

Lines 90 to 95 in 7be5628

	const matchUnpacker = (file: string, dest: string, stat: fs.Stats, tmpUnpackedPaths: Set<string>) => {
	if (this.config.unpackPattern?.(file, stat)) {
	log.debug({ file }, "unpacking")
	tmpUnpackedPaths.add(`**${path.sep}${dest}`)
	return
	}