Updating amplify meta major performance issue

[ErlendHer]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

18.7.1

Amplify CLI Version

12.12.1

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

The "store current cloud backend" step is extremely slow for larger code bases due to some inefficient code in the CLI. For larger projects this step can take several minutes to complete, in our case this step can take more than 15 minutes to complete.

The cause of this issue is found in the update-amplify-meta.ts file, in the moveBackendResourcesToCurrentCloudBackend function. During this step, ALL files from amplify resource folders are copied into the current cloud backend target directory (including node_modules, which in the case of functions with a lot of dependencies can potentially be a very large folder containing many files). After the copying the files over, a check is made to see if any node_modules were copied over (using glob.sync) after which they are deleted.

This is incredibly inefficient. Especially for larger projects this will drastically slow down deploy times.

Problematic code:

const moveBackendResourcesToCurrentCloudBackend = (resources: $TSObject[]): void => {
  const amplifyMetaFilePath = pathManager.getAmplifyMetaFilePath();
  const amplifyCloudMetaFilePath = pathManager.getCurrentAmplifyMetaFilePath();
  const backendConfigFilePath = pathManager.getBackendConfigFilePath();
  const backendConfigCloudFilePath = pathManager.getCurrentBackendConfigFilePath();
  const overridePackageJsonBackendFilePath = path.join(pathManager.getBackendDirPath(), 'package.json');
  const overrideTsConfigJsonBackendFilePath = path.join(pathManager.getBackendDirPath(), 'tsconfig.json');
  const overridePackageJsonCurrentCloudBackendFilePath = path.join(pathManager.getCurrentCloudBackendDirPath(), 'package.json');
  const overrideTsConfigJsonCurrentCloudBackendFilePath = path.join(pathManager.getCurrentCloudBackendDirPath(), 'tsconfig.json');

  for (const resource of resources) {
    const sourceDir = path.normalize(path.join(pathManager.getBackendDirPath(), resource.category, resource.resourceName));
    const targetDir = path.normalize(path.join(pathManager.getCurrentCloudBackendDirPath(), resource.category, resource.resourceName));

    if (fs.pathExistsSync(targetDir)) {
      fs.removeSync(targetDir);
    }

    fs.ensureDirSync(targetDir);

    // in the case that the resource is being deleted, the sourceDir won't exist
    if (fs.pathExistsSync(sourceDir)) {
      fs.copySync(sourceDir, targetDir);
      if (resource?.service === ServiceName.LambdaFunction || (resource?.service && resource?.service.includes('custom'))) {
        removeNodeModulesDir(targetDir);
      }
    }
  }

  fs.copySync(amplifyMetaFilePath, amplifyCloudMetaFilePath, { overwrite: true });
  fs.copySync(backendConfigFilePath, backendConfigCloudFilePath, { overwrite: true });
  /**
   * copying package.json and tsconfig.json to current cloud backend
   */
  try {
    fs.writeFileSync(overridePackageJsonCurrentCloudBackendFilePath, fs.readFileSync(overridePackageJsonBackendFilePath));
  } catch (err) {
    if (err.code !== 'ENOENT') {
      throw err;
    }
  }

  try {
    fs.writeFileSync(overrideTsConfigJsonCurrentCloudBackendFilePath, fs.readFileSync(overrideTsConfigJsonBackendFilePath));
  } catch (err) {
    if (err.code !== 'ENOENT') {
      throw err;
    }
  }
};

Expected behavior

Storing current cloud backend shouldn't take more than 15 minutes.

Reproduction steps

1. Have a repository with a large amount of resources with a node_modules folder
2. Run amplify push where changes have been made to all resources with node_modules (for maximum slowness)

Project Identifier

No response

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey, thank you for reaching out. Marking this as feature request for improvements.

[ErlendHer]

@ykethan I created a PR to fix this: #13821