[Feature Request] Enable parallel installation of libaray

[MichaelSuen-thePointer]

Installation of boost is very slow, each of its sub-package is downloaded and built sequentially, which cause low cpu usage and low efficiency.

command line vcpkg install boost

Proposed solution
If vcpkg can enable package level parallel installation, use vcpkg -j8 like make -j8 to speed up package installaziton, it would be of great help

[PhoebeHui]

@MichaelSuen-thePointer, this is already supported, see

vcpkg/ports/boost-modular-build-helper/boost-modular-build.cmake

Line 197 in 4073d29

-j${NUMBER_OF_PROCESSORS}

You can enable it by setting NUMBER_OF_PROCESSORS environment variable.

[MichaelSuen-thePointer]

@PhoebeHui The -jX option applied on b2 will only enable parallel build of one single boost module right? The bottleneck is not build time of one single module, but the sequential installation of a tons of boost modules.

So if there are mutliple boost module to be installed, the whole process is still like:

download m1-> unpack m1-> parallel build m1-> install m1-> download m2 -> unpack m2 -> parallel build m2 -> install m2

What I wanted is like vcpkg spawn multiple itself and install independent module simultaneously, just like I manually started multiple vcpkg process.

Not parallel build, but parallel download, unpack and install - vcpkg level, not build system level

[strega-nil]

Hmm, in theory this is possible; let me ask the team about it :)

[strega-nil]

Apparently this is a Really Hard Problem. @ras0219 has looked into this before, and we may look back into this in the future.

[AJIOB]

Hello @strega-nil

Do you have any updates about this thread?

[AugP]

We don't support this behavior yet. I do agree that it would be beneficial to support parallel port installations. We are tracking it on our backlog currently, but do not have an ETA.

[bmanga]

Any updates on this? It would be really helpful to have when the number of required packages becomes big :(

[autoantwort]

No. But there are PRs like microsoft/vcpkg-tool#908 to lower the needed time per port

[zufari4]

Any updates on this?

[AugP]

Any updates on this?

We do not have any updates at this time, but we are keeping this request under consideration in our backlog.

[Holt59]

I strongly second this. I have a few boost libraries, most of them headers only, and it takes about 10 minutes to download all the dependencies... It's slower to install Boost libraries with VCPKG than to build the integrity of Boost.

[timmeh87]

+1. I have a project that all together needs 260 packages from vcpkg. I build on a machine with 40 cores. The builds spends more time configuring and linking than anything else, the machine is basically sitting at idle for 5 hours when in theory it could take basically as long as the shortest critical path, which would still be probably an hour or so. But that's still the difference between doing something "today" or "overnight". I wouldn't expect full 100% parallelization all the time on 40 cores but it should be possible to make a set of trees of dependencies and start everything at the bottom in parallel.

It might be hard to decide how many projects can start building at once. Dont want to swamp a small CPU with too many things at once. I do some "extreme parallelization" in my own software and start sometimes 60 parallel builds in parallel, and yeah it does make the machine "a little" unresponsive with many times more build threads than cores, but it does also build everything pretty fast :). You could have a "max parallel projects" parameter given to the user and just let us figure it out. We can set the parallel files per build (the -j parameter) too. So we could tweak it so say have max 7 parallel builds of max 20 parallel files on my 40 core machine. worst case it might try to do 140 files at once but on average its probably going to be lower than that as the configuration and linker stage is always single threaded. It will still be faster in the end and usually when I do my build its a "walk away from that machine" situation anyways cause its still quite annoying to use a machine that unpredictably goes between 0 and 100% cpu use for 5 hours