00:54 <daniels> Venemo, eric_engestrom: can one of you please disable the jobs?

04:07 <Venemo> daniels: which jobs?

08:03 <mupuf> Venemo: that's a valve infra bug... I still haven't figured it out

08:03 <mupuf> I tried again to replicate it on thursday, to no avail.

08:05 <mupuf> daniels: this is a low probability issue. And this runner is for post merge and manual jobs.

08:33 <Venemo> mupuf: it has now failed all my attempts of retrying

08:33 <mupuf> What the heck :o

08:34 <mupuf> I'll reboot my farm in a bit, that may solve the issue

08:34 <mupuf> Let you know when I did it

08:55 <eric_engestrom> would be great if we could move enabling/disabling the farms to an external repo, so that we can do that without having to flush the marge queue (which tends to be quite long these days); is that possible within the gitlab-ci.yml restrictions, or would that mean having jobs always created but they start by downloading a file that tells them whether to `exit 0` right away or actually run the tests?

09:03 <eric_engestrom> mupuf: feel free to cancel https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21851 if you get around to rebooting your farm before the MR lands :)

09:04 <mupuf> This farm isn't used at all for premerge, so no need to disable it

09:05 <mupuf> And yeah, we could easily move these variables to a different repo

09:07 <eric_engestrom> oh ok, so what should I disable then?

09:08 <mupuf> nothing, all the machines fromthis farm are used for post-merge

09:08 <mupuf> I'll reboot it in an hour, when I am done cooking

09:09 <mupuf> (i need to deploy an update, and I don't want to rush it for manual jobs only)

09:10 <eric_engestrom> "all the machines fromthis farm are used for post-merge" you mean VALVE_FARM right? so what do we need to disable for the runners currently down (navi10 from what I can tell)?

09:10 <eric_engestrom> those are used in pre-merge jobs

09:11 <eric_engestrom> navi21

09:14 <eric_engestrom> https://gitlab.freedesktop.org/mesa/mesa/-/jobs/37887739

09:14 <eric_engestrom> https://gitlab.freedesktop.org/mesa/mesa/-/jobs/37887738

09:14 <eric_engestrom> for the two kinds of errors I've seen on them: 500 from some server (I don't know which as I'm not familiar enough with the architecture), and timeout while booting

09:16 <mupuf> Oh boy, sorry yeah, let‹s disable thé farm then

09:16 <eric_engestrom> so VALVE_FARM=offline, right?

09:17 <mupuf> yeah, I'm on it

09:17 <mupuf> sorry...

09:17 <eric_engestrom> ok, thanks :)

09:17 <mupuf> why the fuck would the two farm behave badly at the same time :o

09:33 <eric_engestrom> side-note, but I've been trying to follow through the extends and everything for `virgl-iris-traces` but I still can't see what rule makes it get included on an MR that only touches src/asahi/compiler/* such as https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21674

09:40 <mupuf> eric_engestrom: do you know how to cancel the marge job?

09:40 <mupuf> it seem to be in a limbo

09:40 <eric_engestrom> unassigning make marge move on, or cancelling the pipeline makes marge fail and move on

09:41 <eric_engestrom> and yeah, intel's lava seems to be stuck as well

09:41 <eric_engestrom> that's a lot of infra issues at the same time

09:44 <eric_engestrom> hmm no, intel's farm is running, just too many jobs/not enough runners

09:44 <eric_engestrom> after clicking around I found some jobs that recently starting making progress

09:47 <eric_engestrom> opened everything, looks like only 2 intel jobs can run at a time right now?

09:48 <eric_engestrom> with 15 minutes left I expect it will run over :/

11:24 <daniels> eric_engestrom: which Intel gen? we probably have close to 50 between them, so it’s not that

12:01 <daniels> (though the building they're in seems to lack at least one of power and network)

12:22 <DavidHeidelberg[m]> daniels: Marge is dead?

12:25 <daniels> DavidHeidelberg[m]: marge is fine, but all the SJIC racks are dead

12:25 <DavidHeidelberg[m]> Marge didn't pick up https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21851

12:26 <daniels> oh, iswym, yeah she does appear to be stuck

12:26 <DavidHeidelberg[m]> I deassigned everything, does Marge has timeout when it reloads?

12:27 <daniels> nope, just stuck on !21674

12:28 <daniels> ok, now she's restarted and there's nothing assigned

12:29 <daniels> I'm going to assign !21851

12:30 <DavidHeidelberg[m]> kk

12:40 <eric_engestrom> thanks both for taking over while I was away

12:41 <eric_engestrom> daniels: re: "which intel gen", it was the jobs in https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21698

12:43 <daniels> right, they're all ours rather than Intel's

12:51 <eric_engestrom> ok, I never know which device is where, we really should document that

13:12 <mupuf> Wrt where machines are located, I suggest putting it in the name

13:13 <daniels> eric_engestrom: lima/etnaviv are community, igalia has rpi, google has a3xx/a5xx/a630, valve has most amd, collabora has stoney+raven amd and everything else (intel, panfrost, a618)

13:14 <mupuf> And for us, you can see where it is located in the name: kws, or mupuf

13:14 <mupuf> Kws = Seattle, mupuf = my basement in ~Helsinki

13:29 <eric_engestrom> thanks both, but irc is ephemeral, it would be great to write it down in the docs :)

13:36 <daniels> sure, I’m out for the rest of the day but can review any MR