Heads-up: the build system can now build in parallel #774
Replies: 5 comments 2 replies
-
|
I have to say, it's really freaky watching This M2 Max has 12 cores (8 performance and 4 efficiency) and 32 gb RAM. But it doesn't gain much past 4-5 parallel jobs. It'd be interesting to know what the macOS kernel is actually doing, whether the parallelism is being scheduled on the performance cores or the efficiency cores. |
Beta Was this translation helpful? Give feedback.
-
Yeah, I enjoy that too ;-) I now have a solution running that makes log output usable again when running in parallel. Each shell action redirects its output to a temp file and mamake dumps it all at once when the job finishes. Since mamake itself never runs in parallel (only the shell actions it spawns do that), this creates usable log output while running in parallel. Rule output appears out of order (depending on when they finish) but the dependency blocking ensures it's still a sensible order. Commit coming up soon.
Yeah, that's because many things in this code base depend on each other -- often on iffe feature tests that take a long time to complete; the math.sh test in ksh93 is the worst one for that. So there's only so much potential for parallelisation. When a rule depends on an action that hasn't finished yet, it has to wait for it, there's simply no way around that. To have better parallelisation, we'd have to split those large jobs into multiple smaller jobs that can run in parallel. But that is often very complicated. |
Beta Was this translation helpful? Give feedback.
-
|
It's working perfectly fine on my system (Arch Linux, Ryzen 5600 w/ HT and PBO). GCC 14.1.1: tcc 0.9.28rc mob@e02eec6b: FWIW I also tried switching out mamake-posix_spawn.patchdiff --git a/src/cmd/INIT/mamake.c b/src/cmd/INIT/mamake.c
index 0a577c969..ac9df61ca 100644
--- a/src/cmd/INIT/mamake.c
+++ b/src/cmd/INIT/mamake.c
@@ -156,6 +156,10 @@ static const char usage[] =
#include <sys/wait.h>
#include <time.h>
+#if _POSIX_VERSION >= 200112L
+#include <spawn.h>
+#endif
+
#if !_PACKAGE_ast
#include <errno.h>
#include <signal.h>
@@ -1368,9 +1372,28 @@ static int execute(Rule_t *r, char *s)
{
int stat;
pid_t pid;
-
- if (!state.shell && (!(state.shell = getval(state.vars, "SHELL")) || !strcmp(state.shell, sh)))
- state.shell = sh;
+#if _POSIX_VERSION >= 200112L
+ int spawn_err;
+ posix_spawnattr_t attr;
+ char *script_argv[] = { state.shell, "-c", s, NULL }; /* set at declaration time to avoid malloc */
+
+ /* posix_spawn is supported starting from POSIX.1-2001 */
+ spawn_err = posix_spawnattr_init(&attr);
+ if(spawn_err)
+ {
+ errno = spawn_err;
+ report(3, strerror(errno), "posix_spawnattr_init() failed", 0);
+ }
+ report(-5, s, "exec", 0);
+ spawn_err = posix_spawn(&pid, state.shell, NULL, &attr, script_argv, environ);
+ posix_spawnattr_destroy(&attr);
+ if(spawn_err)
+ {
+ errno = spawn_err;
+ report(3, strerror(errno), "posix_spawn() failed", 0);
+ }
+#else
+ /* fallback for ancient systems without posix_spawn(3) */
pid = fork();
if (pid < 0)
report(3, strerror(errno), "fork() failed", 0);
@@ -1382,6 +1405,7 @@ static int execute(Rule_t *r, char *s)
exit(127);
exit(126);
}
+#endif
/* parent */
/* run job in background if wanted & possible (never for -r) */
if (r && state.maxjobs > 1 && state.strict >= 5 && !state.recurse)
@@ -2866,6 +2890,13 @@ int main(int argc, char **argv)
auto_updprev = search(state.vars, "?", 1);
auto_making->value = auto_prev->value = auto_allprev->value = auto_updprev->value = empty;
+ /*
+ * select the shell
+ */
+
+ if (!state.shell && (!(state.shell = getval(state.vars, "SHELL")) || !strcmp(state.shell, sh)))
+ state.shell = sh;
+
/*
* grab the command line targets and variable definitions
*/
Edit: After testing my old laptop using an AMD V140 (single core, no parallelization), the difference in compilation speed averaged about 1 second faster using |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for testing!
I made mamake count all its fork(2) calls and rebuilt from scratch: That's 995 in total. The following times 995 forks of ksh on my Apple M1 system (a shell background process must always fork as it contains a copy of the parent process): So yeah, it's hardly going to make the difference. :-) Also, it looks like a significant of a fork(2) call runs in parallel, i.e., control is passed back to the caller well before the fork is finished and the child starts. If we don't wait for So, the time cost of fork(2) is that much smaller when running in parallel. |
Beta Was this translation helpful? Give feedback.
-
|
I primarily have low end single board computers and entry level computers that I work with and none of them have hyper threading. All of them have at least 4 CPUs with much smaller amounts of CPU L1, L2, and L3 caches. The new -j parallelism works without issues on all systems tested. However, after running the numbers, I would recommend that the formula be equal to half of available CPUs on entry level CPUs as using more provides little improvement on non-desktop/laptop class CPUs. Attempting to use more CPUs in parallel than hardware supports was slower. I believe the max used CPU cores should be one less than MAX allowing a CPU for operating system, sshd, and etc.
Example of testing on a unit: |
Beta Was this translation helpful? Give feedback.
-
|
New commit: b541c9b mamake now saves up log output by default and dumps it when each job is finished. Pass |
Beta Was this translation helpful? Give feedback.
-
|
I haven't rebuilt ksh since the dynamic lib push went in because I assume this changes things with new libs and maybe headers, like for writing plugins with shlib maybe? I will try building manually and see if I can figure it out from the result. I normally use an ebuild I adapted from gentoo's but don't think there are any gentoo devs that follow this repo closely unless something breaks. |
Beta Was this translation helpful? Give feedback.
-
See ced499a (and follow-up commits) :-)
Now on the dev and 1.0 branches (but not in the 1.0.10 release that is just out -- I need this tested first).
Please test this as described in the linked commit and let me know your thoughts and experiences.
Ping @hyenias, @JohnoKing, @posguy99, anyone else interested
Beta Was this translation helpful? Give feedback.
All reactions