Limit download size of the Pulumi CLI?

[slavafomin]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Hello! It looks like this GHA is downloading a 100+ MB archive that includes binaries for all possible languages/platforms, which is a poor idea from the performance standpoint. It would be better to download only required binaries. This will help to reduce network traffic significantly, especially considering that this download happens on every GHA invocation.

Thanks.

Affected area/feature

CLI download.

[RobbieMcKinstry]

Hi @slavafomin. I'm inclined to say this request isn't high enough impact to justify the heavy-lifting required to make it work. Would you be able to share an estimate of how long this download is taking on average? It's always helpful to get data from the field :-)

1. With the exception of MacOS and self-hosted runners, Pulumi comes installed on GitHub Actions runners by default. Right now, when a user does not specify a specific version of Pulumi to use, this action defaults to latest. It would be better to check whether pulumi is installed and use the installed version instead. That would skip the download phase entirely for a vast majority of runs. If that optimization is enabled, then the suggested optimization of downloading just the required binaries is less impactful.

2. You can launch a GitHub Actions job from a Docker container. For users who want to run, say, a NodeJS Pulumi project, you can boot the Pulumi NodeJS container as the host image for your GitHub Action job and have Pulumi already installed. Admittedly, you won't see any benefit from this approach in conjunction with the Pulumi GitHub Action unless Item (1) is implemented, but if performance is critical, you can execute the CLI directly in a run step without invoking this action. This bullet point isn't very convincing, I confess.

3. Right now, all distributions of Pulumi come with everything (aside from provider plugins which are downloaded lazily). We would have to change our distribution pipeline to offer 6x more release options than we currently do. Each supported platform (windows, darwin/arm, darwin/intel, etc.) would need to build and distribute a separate release for each of our supported languages. That's quite a bit of added complexity. Additionally, we'd need to edit https://get.pulumi.com/ to accept a parameter to download a particular release. Editing this script isn't much work.

4. We'd also need a mechanism for users to specify that they are only building a NodeJS project, so this optimization would likely be opt-in.

Maybe they're write something like:

- uses: pulumi/actions@v3
        with:
          command: preview
          stack-name: org-name/stack-name
          runtime: golang

That would also not pose much of an issue.

In short, I think it would be much more impactful to implement Item 1 described above than to try to optimize the release size. It's a lot more easier to skip the release entirely, and would probably give better results for more people.

What do you think? Would love to hear your thoughts! :)

[RobbieMcKinstry]

I suppose we didn't have an issue for the optimization I described, so I created one.

[RobbieMcKinstry]

I've got to eat my words. I think you're right, the performance improvement is worthwhile, and there's no reason not to implement it.

#837 implements an optimization that skips downloads if a compatible CLI version is present.

Right now, when Pulumi is installed, the default language hosts must also be installed. The Pulumi CLI has a subcommand for installing language plugins individually (which was designed to support e.g. YAML) but this subcommand is broken for the language plugins that are considered default (Node, Dotnet, Python, and Go). So unfortunately we couldn't install just Node even if we wanted to. There are a few Nix packages to make this work if you happen to have a runner with Nix installed.

I think in combination with #804 we'll squeeze as much performance from these optimizations as possible. If we cache plugins, plus skip redownloading the CLI and language hosts where possible, we should end up without any installation steps on a hosted runner with warm cache.

[Frassle]

Right now, when Pulumi is installed, the default language hosts must also be installed. ...

N.B. I expect we'll change the core pulumi distribution to fix this at some point. One day a pulumi cli release will just have the cli in it, and dynamically fetch language plugins the same as provider plugins. Probably not worth trying to do all the effort here to work around that for now.

[RobbieMcKinstry]

Yes, I totally agree, that's the path forward in the long run. But it'll take a bit to get there unfortunately.