15:39 <bentiss> FWIW, there is something wrong on https://gitlab.freedesktop.org/mesa/mesa/-/jobs/64290113 -> this is the only one job running ATM on m3l-31 and it sees only 8 cpus, but the global load of the machine is 120. So either it still manages to see 64 cpus and spawns that many jobs, either there is a hardcoded value somewhere that is wrong

15:41 <bentiss> in case it matters, it was the fedora-release job. It would be nice if somebody could double check that job definition.

15:53 <MrCooper> all Mesa build jobs use a ninja wrapper script which runs '/usr/bin/ninja -j${FDO_CI_CONCURRENT:-4} "$@"'

15:53 <MrCooper> the fedora-release job enables LTO though, maybe LTO linker stages spawn more threads

15:56 <MrCooper> I suspect it's mold, it spawns up to 32 threads by default

15:58 <psykose> the linker itself spawns a lot of linking threads but that is basically instant for linking itself and wouldn't cause that load, lto is the most likely reason

15:58 <MrCooper> eric_engestrom DavidHeidelberg: ^ Mesa should probably set the environment variable MOLD_JOBS=1 and pass --thread-count=${FDO_CI_CONCURRENT:-4} to mold

15:58 <psykose> wish ninja there is no jobserver

15:58 <psykose> with*

15:58 <MrCooper> psykose: LTO means most of the compiling work happens in the linker

15:59 <psykose> i know what lto is

15:59 <psykose> does fedora-release use gcc or clang thinlto

15:59 <MrCooper> the former

15:59 <psykose> what does it pass for -flto? =auto?

16:00 <psykose> if that then it would just default to cpu count with no jobserver

16:00 <MrCooper> anyway, if there are multiple mold invocations each spawning up to 32 threads doing LTO work, that could certainly explain the load

16:01 <MrCooper> psykose: just enables meson b_lto

16:02 <psykose> that does use =auto

16:02 <psykose> you would want b_lto_threads to change it

16:03 <psykose> but backend_max_links=2 is probably a better solution in my experience

16:03 <MrCooper> b_lto_threads defaults to 0, with description "Use multiple threads for Link Time Optimization"

16:04 <psykose> yes, =0 maps to auto

16:04 <MrCooper> anyway, seems like you know how to solve it, I'll get back to other things then :)

16:05 <psykose> i know of a lot of workarounds that might work sure, but they're all workarounds, the fundamental issue is a bit unsolvable :P

16:15 <psykose> derusted my memory a bit: yeah, with -flto=auto the linker invokes gcc lto-wrapper and that lto-wrapper emits a makefile with -jdetected-cpu-cores, mold --threads=1 doesn't affect that as expected

16:15 <psykose> just makes mold's actual linking use 1 thread

16:18 <psykose> you could do backend_max_links=1 (this prevents ever going above 8 load), you could do backend_max_links=2 and set b_lto_jobs=jobs/2 (not sure if this is better), you could cap to 2 jobs and accept the load goes over the configured -j

16:18 <psykose> i usually pick the last option but ymmv

16:19 <psykose> actual granular per-job usage up to a -j is not possible here unless ninja grew a make jobserver to complement the emitted make invocation gcc lto-wrapper does where it can act as a jobserver client with =auto

16:19 <psykose> (and for clang thinlto even that is not supported, you just always use a configured threadcount)

16:20 <psykose> (cap to 2 jobs = backend_max_links=2, since i realise now that is ambiguous)

16:26 <psykose> i think for gcc lto and mold i would pick 1 link tbh because it's not as prone as thinlto with lld to sit on 1 thread near the end for 10 seconds, so here https://img.ayaya.dev/6P4zPJFrGVXG