Throttling parallel tasks

[IcedQuinn]

Is there a mechanism (somewhere or planned) to support limiting how many tasks run in parallel?

Ninja supports creating "pools" of varying sizes which tasks can be placed in to. This allows for example one C++ file to compile per core of the computer except for an amalgamation file (like sqlite) or a huge link job (like clang, firefox, ..) where only one job is allowed to run across the whole system--but only for specific troublesome jobs--not setting the whole build to one core or so.

This is probably not super relevant if you are using remote workers but its a neat (if not frequently mentioned) feature Ninja has to control the job graph.

[Tatskaari]

We have a set number of tasks that can run in parallel at the moment. At Thought Machine it's set to 250. Each task is given a weight of 1 but I could see us saying this task uses 2 cores so it takes up 2 task queues. It would be kinda tricky to implement ontop of the current tasks queue, but re-architecting that to use weighted semaphores has been on the cards for a while now.

What's the motivation here though?

[IcedQuinn]

What's the motivation here though?

This allows for example one C++ file to compile per core of the computer except for an amalgamation file (like sqlite) or a huge link job (like clang, firefox, ..) where only one job is allowed to run across the whole system--but only for specific troublesome jobs--not setting the whole build to one core or so.

some build jobs (linking a huge c++ program) are bigger than others (building individual c++ files.) standard parallel limits require you to either overload the system or turn off parallelism due to just one job hogging the entire thing.

[Tatskaari]

Yeah, conceptually, I think it could be an interesting idea, but it's a lot of work because of the way we've implemented build tasks. The vast majority of Please build actions are single threaded, including linking (cc_binary etc.), so it seems hard to justify from my perspective. I think llvm (i.e. lld) might have a flag to parallelise some parts of it these days but that's not really the norm.

[IcedQuinn]

The vast majority of Please build actions are single threaded,

Not sure how that matters.

If a box has 4gb of memory an it takes 200mb RAM to compile many small files—the machine can build twenty at a time. If the link job takes up 3.5gb the machine can't do anything else. That's why Ninja (which was written to compile browsers initially) allows you to set the parallel limit to one thing for cheap jobs and restrict the whole build to just one job for that specific task that floods the machine with work.

It matters more when there is a single build node of moderate means. Not quite as much if you have a dozen build nodes that is individually only doing one thing.

For example I once had a script that would build an underlying C library by just calling in to the makefile within the please container. If you set the parallel limit to one (so it did not try to build six entire libraries that way at once) then parts of the repo that were using please rules wouldn't get run in parallel. etc.