00:05 <DavidHeidelberg[m]> zmike: btw. this must be some regression, I saw this one on my MR too, but I though it's cosmic dust, but probably not.

00:05 <zmike> :/

00:05 <zmike> I'll retry then

00:05 <DavidHeidelberg[m]> I looked into list of previous MR, but nothing extra suspicious

00:06 <DavidHeidelberg[m]> the build jobs are regression for sure, the t860 just networking issue on LAVA farm I assume (reported to them)

00:06 <zmike> yeah I assumed for the t860

03:07 <anholt> zmike: I just got hit by util_tests, too. seems like a regression todayish.

07:31 <MrCooper> anholt zmike: it's https://gitlab.freedesktop.org/mesa/mesa/-/issues/9222

12:31 <karolherbst> mhhh... something up with CI? https://gitlab.freedesktop.org/mesa/mesa/-/jobs/47493714

12:33 <MrCooper> karolherbst: see https://gitlab.freedesktop.org/mesa/mesa/-/issues/9222

12:34 <karolherbst> ohh.. mhh. I should have clicked that link, but it sounded like something different...

15:23 <eric_engestrom> I think this ^ is a problem with fdo-equinix-m3l-24 specifically, as I've only ever seen 24 fail (no other runner) and I've never seen 24 pass (although I don't have much of a sample set for that)

15:23 <eric_engestrom> daniels, bentiss: ^

15:45 <MrCooper> eric_engestrom: the jobs referenced in the issue ran on fdo-equinix-m3l-22

16:14 <eric_engestrom> MrCooper: indeed, I hadn't looked at the ones in the issue. that was 2 months ago though, and 22 looks like it doesn't have this issue anymore (I've seen several pass on that one)

16:14 <eric_engestrom> the admins will know what has or hasn't changed there

16:16 <anholt> we expecting a lot of 500s from the registry right now? been having a lot of truoble this morning.

16:18 <MrCooper> eric_engestrom: out of curiosity, did any of the passing ones reuse an existing container?

16:18 <eric_engestrom> container? I'm not sure what you mean/how to check

16:19 <eric_engestrom> (or did you mean "reuse a git clone"?)

16:20 <MrCooper> I guess that's mostly equivalent, container as in execution context

16:21 <MrCooper> there's "Reinitialized existing Git repository" in the job output in that case

16:25 <eric_engestrom> MrCooper: yup, https://gitlab.freedesktop.org/mesa/mesa/-/jobs/47508855/raw for instance ran on 22 and has "Reinitialized existing Git repository"

16:26 <MrCooper> k, would also be interesting if there are any failures without that, not sure it's likely to be related though

16:27 <koike> sima, airlied, robclark (and others), I'd like to check with you how would you prefer to move forward with the ci patch

16:27 <eric_engestrom> also, this is not related to in-repo CI config, as nothing changed on 23.1 for that and it started failing between thursday (10) and today

16:28 <koike> sima, airlied, robclark (moving to #dri-devel)

16:30 <MrCooper> eric_engestrom: I'm wondering if it might be related to leftover files from previous (possibly failed) jobs

16:31 <MrCooper> eric_engestrom: the issue was filed on June 18th

16:45 <eric_engestrom> yeah, but it was filed about runner 22 which today works fine

16:45 <eric_engestrom> so I think whatever was the issue back then has been fixed now, and potentially the same mistake was done on runner 24 in the last few days

16:47 <eric_engestrom> MrCooper: all the failing jobs I could easily check also have "Reinitialized existing Git repository" in the log, so that's not the difference

16:50 <MrCooper> that means the hypothesis it's related to that isn't falsified yet

16:50 <MrCooper> it could be a required but not sufficient condition

17:12 <dcbaker> I'm still having a ton of issues with CI on the 23.2 branch, I can't get enough of a run in to feel confident making releases, or even pulling patches. Basically everything is failing on lava jobs like this: https://gitlab.freedesktop.org/mesa/mesa/-/jobs/47519904

17:15 <DavidHeidelberg[m]> dcbaker: let me report it :( but probably only retry is solution (if it helps, at least until tomorrow)

17:23 <robclark> dcbaker, DavidHeidelberg[m]: that is the same issue I've been hitting.. daniels said something about collabora lab caching proxy not working or something along those lines

17:27 <zmike> can someone reassign this to marge whenever it stops being broken https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24673

17:29 <DavidHeidelberg[m]> zmike: I'll make a note :) sorry for complications

19:02 <anholt> bentiss, daniels: I've been seeing a lot of 500 errors from the registry all morning. is this expected right now?

19:59 <daniels> eric_engestrom: nothing ever changes on the runners; they’re immutable for their lifetime. that’s why we’re up to -24, because we create new ones rather than change old ones. the container environment is very sealed as well, so it is definitely a test problem rather than a host problem

19:59 <daniels> anholt: probably … the migration to new infra is still incomplete

20:00 <daniels> DavidHeidelberg[m], robclark, dcbaker: yep, somewhere along the line nginx seemingly silently stopped caching

20:00 <daniels> DavidHeidelberg[m]: maybe chase Doug in the thread in the lab channel?

20:01 <DavidHeidelberg[m]> done :) (I'll just add the link)

20:06 <anholt> how many a660 boards do we have?

20:23 <DavidHeidelberg[m]> anholt: currently 7 available, total 9

20:24 <DavidHeidelberg[m]> *just total 9 devices available

21:48 <DavidHeidelberg[m]> Do we know if x86_64 or aarch64 is more utilized? (if I can decide where to run, what to prioritize). I assume there is less aarch64, but less utilized, but conformation would be nice :)

22:36 <robclark> DavidHeidelberg[m]: aren't there some ci stat grafana thing?

22:39 <DavidHeidelberg[m]> robclark: I didn't found any board with it, I see transfers and stuff, but not usage

22:40 <robclark> hmm, ok I thought there was something w/ # of jobs

23:56 <daniels> aarch64 is much more free