00:27 <daniels> thankyou!!

07:04 <bentiss> OK, I've re-enabled the new runners at Hetzner this morning. I've fixed the issue with the jobs not setting up properly by disabling the docker cache. Priorities are up again

08:00 <bentiss> Just saying, it would be nice if someone writes a MR to assign the priority based on the user (marge gets priority high for instance). I really won't have the time to work on that as I still have the registry to finish migrating

08:02 <daniels> awesome! thanks

08:32 <hakzsam> just noticed something weird. Since the migration, it seems gitlab no longer delete source branches when a MR is merged, although "Delete source branch when merge request is accepted." is enabled

08:39 <dj-death> hakzsam: yeah noticed the same thing

08:40 <hakzsam> ah, interesting

08:57 <MrCooper> I see the same in GNOME gitlab, so looks like a gitlab regression

14:02 <hakzsam> are all the runners up now?

14:02 <daniels> yeah

14:03 <hakzsam> very cool

16:07 <daniels> (and just as we finally get almost all of CI back, someone at our upstream ISP gets brave after Friday lunch and decides we need more routing loops in our life)

16:42 <bentiss> Looks like the new bare metal runners are working fine this time, I'll go release the ones from Equinix

16:46 <bentiss> and done... I had a little bit of sweat to ensure I wasn't removing the wrong servers :)

16:49 <pinchartl> bentiss: it's a horrible feeling when you know there's a risk you may be about to make a horrible mistake

16:50 <pinchartl> I read a story once about people standing around a computer, staring at the screen showing a ctrl-C in the middle of a rm -rf / run as root on a production machine

16:52 <alanc> I noticed the failure to delete branches from MR's just before the migration, when I was doing a last batch of merging before the downtime, so it's not new to the post-migration site

18:10 <Venemo> good afternoon guys

18:10 <Venemo> nice job on the gitlab migration

18:11 <Venemo> just 1 question, why is marge bot so slow now? does it have to do with the migration, or is it just that everyone wants to merge a lot of things at the same time?

18:11 <daniels> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/?assignee_username=marge-bot

18:11 <Venemo> I've already checked that

18:12 <Venemo> it doesn't answer my question

18:12 <daniels> 14 MRs in the queue; if you assume a 20-30min turnaround time, then the time from placing an MR in the queue to it getting merged is around 4.5-7h

18:12 <daniels> or were you asking a different question?

18:13 <Venemo> so, it's due to too many MRs

18:13 <daniels> (it was draining much more slowly this morning due to people having managed to sneak in various regressions, so a lot of the MRs were running long with retries until the expectations all got fixed)

18:13 <Venemo> btw, isn't there an upper limit on how long a CI job can run?

18:13 <daniels> right - just look at the queue, mentally divide the number of entries by 2 or 3, and that's the number of hours you can expect to wait

18:14 <daniels> yeah, we have timeouts all over the place

18:14 <Venemo> what is the upper limit?

18:14 <daniels> grep -r timeout: .gitlab-ci* **/*.yml

18:14 <daniels> varies

18:14 <Venemo> wasn't it 15 min or do I misremember?

18:15 <daniels> https://docs.mesa3d.org/ci/

18:15 <daniels> 'A typical run takes between 20 and 30 minutes', and there's a lot of explanation in later sections about the timeouts we use and how they're calculated

18:16 <Venemo> it looks to me like the currently in progress job has been running for 40 minutes now: https://gitlab.freedesktop.org/mesa/mesa/-/jobs/73555351 and in the meantime the rest of the CI is unused when it could already start running the next job

18:16 <daniels> realistically they're set to values which are appropriate for the job, to allow occasional spikes (i.e. don't kill a job just because the network was having a slightly bad day) but not anything too egregious

18:16 <daniels> yep, looks like virpipe-on-gl is missing a timeout

18:18 <Venemo> It says: Timeout: 1h (from project)

18:20 <Venemo> it's really strange

18:22 <Venemo> anyway, thanks daniels, I guess I'm not that much in a hurry to get anything merged

18:23 <daniels> yeah, that job's just missing a timeout is all

18:23 <daniels> np, hope your stuff lands soon

18:27 <Venemo> I suppose a lot of stuff got queued up last week

18:37 <daniels> yeah, and the whole run from Wednesday through to this morning really sucked as we had a bunch of instability (no-one's local caches were caching, loads of tests had become flaky either environmentally or due to being pushed in when most tests were disabled, network issues, new runners needed tweaking to work reliably), so there's a lot of backpressure from there

18:47 <daniels> oh, and if you want quicker CI, tbh the best thing you can do right now is to stop people writing more compilers to add to the build, or stop writing things in Rust :P

19:00 <demarchi> daniels: I've seen some messages about moving the host handling ssh and pointing to https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/2076 - but it's not clear if it's something to be done now, in future, or if the old one will continue to work

19:00 <demarchi> daniels: for the kernel trees in drm-tip, we'd probably want to change the manifest to the new ssh host if this is going to change

19:04 <gchini> Hi, I just tried to change my e-mail address but I did not receive the confirmation e-mail. Should this already work or is there still something open from the migration?