08:59 <Venemo> error 503 here https://gitlab.freedesktop.org/mesa/mesa/-/jobs/35789968

09:34 <MrCooper> bwidawsk: if you need anything beyond just installing a fixed set of packages, my advice would be to handle everything in the FDO_DISTRIBUTION_EXEC script and don't bother with FDO_DISTRIBUTION_PACKAGES

09:36 <MrCooper> bwidawsk: the templates automatically do the autoremove purge dance at the end now, that could be removed from westong

09:37 <MrCooper> (ugh, I was reaching for backspace, but hit enter instead :)

09:53 <daniels> MrCooper: hm? it’s removing ephemeral packages that we use only in the container build stage

09:53 <daniels> same as Mesa

09:54 <MrCooper> ah right, templates just do apt-get autoremove

10:32 <mupuf> Venemo: works for me. Does it for you?

11:34 <Venemo> mupuf: I dunno, I just noticed this problem on an overnight run

11:34 <Venemo> I reassigned to the bot again

11:35 <mupuf> Venemo: right. This is a temporary failure, so yeah, annoying

11:46 <Venemo> could there be some different handling of these kinds of failures vs. when the tests really fail?

11:51 <daniels> not really; we've already stuffed in as many retries as we can

11:51 <Venemo> I mean instead of "CI failed" I'd like to see a different message

11:53 <Venemo> currently "CI failed" is 50%-50% split between: "CI crapped itself because of a networking error so it didn't even run any tests" vs. "CI actually ran the tests but some tests failed because I wrote crap code"

11:53 <daniels> you mean a different message from marge?

11:53 <Venemo> yes, that's exactly what I mean

11:54 <daniels> being worked on

11:54 <Venemo> "CI was unable to run properly" vs. "CI ran tests successfully but some of them failed" or something along those lines

11:56 <MrCooper> how would Marge be able to tell the difference? (Or is the plan to replace her with ChatGPT? ;)

11:56 <Venemo> same way I can?

11:56 <Venemo> I don't understand the question

11:56 <MrCooper> are you volunteering then? ;)

11:57 <Venemo> for what?

11:57 <MrCooper> not sure what difference it would make anyway, the MR can only be merged if CI passes

11:58 <Venemo> in case I wrote crap code, I need to investigate it and fix it. otherwise I just reassign to marge

11:58 <daniels> ManMower: the CI daily reports already do that kind of classification; they break down whether failures are due to infrastructure / hung machine / timeout / failed tests. we've been working on cleaning up and generalising that code, combining it with some of the ci-uprev and ci_run_n_monitor so we can reuse the same stuff across the board and make marge able to tell you more without having to go read the log.

11:58 <daniels> in the meantime you have to click through and read

11:58 <Venemo> usually what I do is just always reassign to marge and if it fails 2-3 times then I look at the actual logs

11:59 <Venemo> it's really tedious to find out anything from those logs, so I want to avoid that

11:59 <MrCooper> that's wasting infrastructure resources

12:00 <Venemo> yes, that's why I made the suggestion above

12:01 <daniels> yeah, don't do that

12:01 <Venemo> if it were clear what happened, I wouldn't

12:01 <daniels> the 503 seems pretty clear to me

12:01 <daniels> we've been working on cleaning up the logs as well to make them more immediately valuable

12:02 <daniels> but if you're just smashing resubmit 3 times before you even try to look, it's hard to take complaints about the merge queue being too long seriously

12:02 <Venemo> it's not just me, everyone does this

12:03 <daniels> I wouldn't say everyone tbh, but yeah, there certainly are some

12:03 <Venemo> I'm exaggerating, of course

12:03 <daniels> I've watched people smash MRs at Marge like 5 times in a row before I had to tell them that their code didn't even compile so could they please stop

12:03 <daniels> it's hard to know what to do when someone cares that little

12:03 <Venemo> about 90% of the time, I already test our stuff in the valve infra, so I rarely assign anything to marge that I didn't already test

12:04 <Venemo> it is then very frustrating to see a failure from something that just passed 5 minutes ago

12:04 <Venemo> also most of them time even before going to the valve infra I run tests locally as I see reasonable

12:06 <daniels> I don't know what else to tell you. people are working hard to fix it, but when the average level of care is 'I can't even be bothered reading the logs so I just make it try again', then it's not exactly easy

12:06 <Venemo> no that's not what I'm saying

12:07 <Venemo> I'm saying that after all that effort spent testing the thing BEFORE submitting them to the CI, IMO it's more likely to fail due to a network error than be an actual test failure

12:07 <Venemo> also, I appreciate the work of everyone involved

12:07 <daniels> by far the most common reason for things to fail is flakes, which is exacerbated by people just smashing retry without reading logs or doing anything about any observed failure

12:08 <daniels> hence why we've been working on automated systems to find and categorise flakes and just update expectations automatically

12:08 <Venemo> I don't doubt that but I personally rarely see flakes these days

12:10 <Venemo> maybe we could allow the CI jobs to return a custom error string that indicates why the job failed? then each job would be able to give some useful info

12:10 <daniels> yeah, that’s exactly what we’ve been working on

12:10 <daniels> but flakes are the biggest issue (and we also have other priorities) so we’ve been working on that first

12:11 <Venemo> all right then that's great news

12:11 <Venemo> thanks for doing this

12:17 <daniels> np!

17:07 <bwidawsk> MrCooper: thanks for the suggestion. I'm looking now at doing multiple builds for different feature sets in a Rust crate. Using cargo I think it makes sense to do these all as a single build target so cargo doesn't have to do a full build for different feature sets. Curious if you have advice there.

17:12 <bwidawsk> (it would be preferable for me to run them as different jobs, but I believe that's wasteful)

17:16 <bwidawsk> meh, I think I'll leave well enough alone until it becomes a problem

18:53 <bwidawsk> the runner seems to eat output from any scripts invoked by EXEC

18:53 <bwidawsk> Am I doing something wrong?

19:21 <bwidawsk> So it'd be pretty nice if I could have FDO_DISTRIBUTION_CARGO_PACKAGES or some such

19:21 <bwidawsk> then I don't have to reinstall a bunch of crates every run

19:21 <bwidawsk> For whatever reason I can't seem to get them to be permanently there using a script in EXEC

19:48 <bwidawsk> oh, I needed to bump the tag

19:48 <bwidawsk> ignore me

22:26 <pobrn> Hi, sorry for the banal question... but has the font changed on gitlab.freedesktop.org? I could swear it looked different a couple days(?) ago, but maybe something on my end has changed (or I am just imagining it...).

22:27 <vyivel> pobrn: yes, gitlab was updated and the new version has different fonts

22:37 <pobrn> Thanks for the reply. Interesting, but I don't think that affects me as of yet, in my case Firefox simply started choosing Noto Sans instead of Segue UI.