Yarn fetches all packages on every build (cache issue)

[aaronadamsCA]

Despite a 289 MB cache being restored at the start of each build, Yarn v2 fetches all but 4 packages during every build.

Build directory: packages/studio

packages/studio/netlify.toml:

[build.environment]
  NODE_ENV = "production"
  NETLIFY_USE_YARN = "true"
  YARN_FLAGS = "--immutable"

Log:

2:00:17 PM: Build ready to start
2:00:19 PM: build-image version: 653805ca4a64301556e56dc4b321ef8fc20cbb7c
2:00:19 PM: build-image tag: v3.8.2
2:00:19 PM: buildbot version: ccda11705f87435917703d79785086e961e5d73c
2:00:19 PM: Fetching cached dependencies
2:00:19 PM: Starting to download cache of 288.9MB
2:00:20 PM: Finished downloading cache in 1.206042171s
2:00:20 PM: Starting to extract cache
2:00:29 PM: Finished extracting cache in 9.180126006s
2:00:29 PM: Finished fetching cache in 10.446618482s
...
2:00:40 PM: Started restoring cached yarn cache
2:00:40 PM: Finished restoring cached yarn cache
...
2:00:44 PM: Yarn workspaces detected
2:00:44 PM: Started restoring workspace packages/scripts node modules
2:00:44 PM: Finished restoring workspace packages/scripts node modules
2:00:44 PM: Started restoring workspace packages/studio node modules
2:00:44 PM: Finished restoring workspace packages/studio node modules
2:00:44 PM: Started restoring workspace root node modules
2:00:44 PM: Finished restoring workspace root node modules
2:00:45 PM: Installing NPM modules using Yarn version 2.4.2
2:00:46 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
...
2:00:47 PM: ➤ YN0000: ┌ Fetch step
2:01:32 PM: ➤ YN0013: │ 4 packages were already cached, 1532 had to be fetched
2:01:32 PM: ➤ YN0000: └ Completed in 45s 880ms
...
2:02:50 PM: Caching artifacts
2:02:50 PM: Started saving workspace packages/scripts node modules
2:02:50 PM: Finished saving workspace packages/scripts node modules
2:02:50 PM: Started saving workspace packages/studio node modules
2:02:50 PM: Finished saving workspace packages/studio node modules
2:02:50 PM: Started saving workspace root node modules
2:02:50 PM: Finished saving workspace root node modules
...

IMO the "right" way to handle Yarn caching is to cache .yarn/cache (or wherever your yarn config get cacheFolder is), and let Yarn rebuild node_modules/ on each build. You want to cache its cache between builds, not its outputs.

[julienw]

I believe that the problem is that .yarn/cache only grows, so if they were caching it the cache would only grow.

[julienw]

Actually this may have been done some days ago in #595.

[JGAntunes]

Hey @aaronadamsCA 👋 thanks for raising this issue.

The problem is actually a Yarn 2 support issue, related with #319. The custom cache directory we rely on for yarn (and configure via the cache-folder property) does not work when using yarn 2.

I've created #612 and linked your issue to it. If it's alright with you I'm going to close this issue and we can follow up in there 👍

I believe that the problem is that .yarn/cache only grows

I don't believe that will be the case here @julienw. In fact the resulting cache artifact is likely to not contain any .yarn/cache contents due to what I've mentioned above ☝️