11:00 <emersion> daniels, do you want help debugging this Go code?

11:20 <daniels> emersion: if you had some time to help look into nginx+workhorse that would be good thanks - I was out for the latter half of last week, and going to be stuck catching back up with work for most of the next couple of days (great timing to have a personal thing I really can't move for most of the day today ...)

11:20 <daniels> I can shoot you some instructions and get you sorted with access when I get back later tonight

11:20 <emersion> yup, can do!

11:22 <daniels> thanks!

11:22 <daniels> really appreciate it

11:23 <kennylevinsen> if needed I can also volunteer some time on this issue

15:49 <bentiss> daniels: so that's weird. I moved all gstreamer projects yesterday, and I have no 504 for them expect 6 for a non existant project "/gstreamer/desktop-file-utils.git" The weird part is why is it waiting for 1800 secs to answer a 404...

17:34 <bentiss> FWIW, I am moving 10 more namesapaces from gitaly-3 to gitaly-1: pulseaudio, pipewire, libinput, mobile-broadband, libfprint, polkit, cairo, dbus, geoclue, upower. So far, gitaly-1 is holding, so if we start seeing failures in gstreamer, NetworkManager or any of these projects then there is one culprit among those 10 namespaces

17:37 <__tim> fontconfig would be useful too fwiw, in case there isn't a fixed order (as our ci pulls it) :)

18:19 <nirbheek> also libnice :)

19:31 <ndufresne> not sure what this is about, getting very slow repo update, and 504 sometimes on fontconfig

19:31 <ndufresne> I guess related to the previous discussion ;-D

19:31 <bentiss> alright, it seems to hold for now. I am adding fontconfig and libnice to the gitaly pod that doesn't seem affected by the bug

19:32 <bentiss> ndufresne: yeah, I think we have a blackship (or maybe more than one) in the infra and it completely messes up the gitaly servers

19:32 <bentiss> black sheep

19:38 <ndufresne> We should bleach these sheep

19:39 <bentiss> on that note, I should mention that our last successful backup was 13 days ago... this ins not good :(

19:39 <bentiss> especially given that I think we prune backups after 7 days

19:49 <__tim> and freetype too please :)

19:49 <__tim> user forks are in their separate namespaces I presume?

19:55 <bentiss> nope, when I move a namesapce, to ensure git dedup I have to take all the forks too

19:56 <bentiss> FWIW, only freetype/freetype-demos is on gitaly-3, so not sure it'll help

20:09 <bentiss> daniels: the backup was failing because goldsteal/xkeyboard-config was referring to a non existant pool. I manually copied the pool from gitaly-3 to gitaly-1 with the expected name and now git seems happier

20:09 <bentiss> I wonder if we have other issues like that

20:20 <daniels> bentiss: *blink*

20:20 <daniels> that was one of the ones which was failing to migrate when I did gitaly-3 in the first place

20:20 <daniels> so I wonder if it did actually complete and then that just never got recorded?

20:23 <bentiss> daniels: well, that one was pretty busted, the db thinks it doesn't have a pool when the alternate file clearly mentioned one

20:23 <daniels> urgh

20:24 <bentiss> anyway: for the analysis if gitaly-1 is happy, I use elastic and look for 504 errors, they are showing up there, so if 'GET /gstreamer' is showing up, that means gitaly-1 is now busted

21:57 <anholt> oof. didn't even get the pipeline started in mesa marge's 1 hour timeout.

22:19 <daniels> anholt: which pipeline?

22:19 <daniels> emersion: you've got an Equinix Metal invite and https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/413 has everything I could think of off the top of my head

22:20 <anholt> was https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/481726

22:30 <daniels> anholt: ouch.

22:30 <daniels> so part of it is that both NM and GSt have some exceptionally long-running integration test jobs running

22:31 <daniels> and then most of the rest of the capacity went to failing to clone libqmi

22:31 <anholt> some jobs at the time were backed up on git fetches, but some of it was just it looks like we only have 3 runners these days?

22:31 <daniels> bentiss: do you still have the toolbox open & able to pull libqmi?

22:31 <daniels> *over to gitaly-1

22:31 <daniels> anholt: yeah, we can bring back the fourth if shared capacity is a consistent thing

22:32 <anholt> it's certainly been an issue for the last week-ish, but that's mixed up with the git fail

22:33 <anholt> though llvmpipe/virgl jobs have been the long pole in mesa ci for a while I think. more jobs than we have runners, even if a job should be <10 min.

22:35 <emersion> daniels: thanks! will look at all that tomorrow

22:36 <daniels> emersion: np, thanks to you :)

22:36 <daniels> anholt: yeah, probably time to combine some of them?

22:37 <bentiss> daniels: will try

22:38 <anholt> I guess we could combine to reduce the overheads, but really we have more work we want to get done on testing sw rasterization than we have time to complete it

22:39 <anholt> would sure be cool if we could get a relevant company to just sponsor some big dedicated Mesa runners. (sigh)

22:39 <bentiss> daniels: libqmi is already on gitaly-2

22:40 <daniels> bentiss: urgh, I just saw a bunch of jobs stuck on cloning that :(

22:40 <bentiss> there are 6 504 errors on libqmi since the move

22:41 <daniels> anholt: yeah ok, if it's a problem then we can ask Equinix if we can take more capacity; if they say no then Collabora can foot that bill

22:46 <bentiss> anholt: FWIW, the fact that we are seeing timeouts right now on many projects and that this timeout is 30 min is heavily impacting the runners availability :/

22:46 <bentiss> we are also still one runner down that we use for MinIO caching of the repos

22:48 <bentiss> daniels: sigh -> Backup::Error: gitaly-backup exit status 1 ... (goldsteal/xkeyboard-config): manager: write bundle: *backup.FilesystemSink write: write file \"/srv/gitlab/tmp/backups/repositories/@hashed/1b/0f/1b0fa6c4c967f1c21f9c23ecc41ef8a27528a7be52b29d65d47123f5ac18d5be.bundle\": rpc error: code = Internal desc = cmd wait failed: exit status 128, stderr: \"fatal: bad object

22:48 <bentiss> refs/tags/v_1_5\\n\"\n"

22:50 <daniels> bentiss: well, that at least explains why that repo didn't move ... !

22:51 <daniels> bentiss: I think I'll just nuke that repo then email him and ask him to re-push?

22:57 <bentiss> daniels: I was just about to fix it...

22:57 <daniels> ah

22:57 <daniels> well, that's one way to fix it

22:57 <bentiss> my git bundle was just containing master, so I created a new one with all the tags

22:58 <bentiss> but anyway, should be better now

23:01 <bentiss> OK, I'm done for today, I think I said that I was going to bed more than an hour ago :(

23:02 <bentiss> daniels, emersion: maybe a good testbed for Simon would be to do the gitlab security upgrade tomorrow morning?

23:04 <daniels> oh yeah, that's a good shout

23:04 <daniels> though I'd have to back out the custom Gitaly bit first

23:04 <daniels> bentiss: yeah, I'm hoping I can sleep tonight rather than being kept up by my sinuses trying to break free of my skull

23:05 <daniels> (hence not really doing anything more intelligent than braindumping docs)