07:57 <bentiss> It's Friday morning and we have a gitlab security update pending... So how about I do it now?

08:13 <bentiss> FWIW, the release notes mention a minor bump of postgresql, so I'll also have to do it once the current gitlab migration ends

08:30 <daniels> bentiss: go for it - it’s Easter so should be quiet

08:31 <bentiss> hopefully :)

08:32 <bentiss> right now the gitlab migration is still running... hopefully a restart/upgrade of the db will help for the next one

09:49 <emersion> IPv6 is broken again

12:26 <__tim> have sen multiple issues now with merge request merges failing but the commits actually successfully having been added to the main branch fwiw

12:27 <__tim> manually pressing the merge button sorts it after it spins for a long time

12:28 <__tim> (example: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1433 just in case you need something to look for in the logs)

12:59 <bentiss> __tim: the migration of the db didn't completely went through, so this is not overly surprising

12:59 <bentiss> re-lauching it ATM

13:01 <bentiss> sigh... main: == 20240212031520 SyncIndexForPCiBuildsPart3: migrated (12752.2315s) ========== that one took a little bit longer than the 1h timeout

16:23 <alanc> Hopefully we don't have sshd installed in too many CI containers, because I know we have xz-utils installled in many of them (for "make distcheck" or "meson dist") and at least some are built using Debian testing or unstable: https://www.openwall.com/lists/oss-security/2024/03/29/4

16:25 <mingdao> Does your sshd link to lzma?

16:25 <mingdao> Ours doesn't.

16:26 <mingdao> xz-utils needs to be downgraded, probably.

16:31 <alanc> apparently that depends on whether patches have been applied for systemd support in sshd

16:32 <alanc> I don't have access to scan the CI containers to see which do or don't - that's a challenge for the gitlab admins

16:52 <bentiss> alanc: our runners are pretty much trashable anytime, and are gate-kept by runner_gating.sh. So if we get attacked through this way, that mean we put too much trust in one person and can probably talk to them

16:53 <bentiss> also you don't need such backdoor to gain access to them, it's fairly easy to escape the container jail given that they ar privileged

16:53 <bentiss> but thanks for the heads up!

16:53 <bentiss> side note: the migration still didn't complete, so that's a tad worrying

16:57 <alanc> I wasn't worried so much about our users doing bad things as much as someone external being able to break in that way, but I suppose they'd also have to be lucky enough to hit the small windows when each container is running, since for most projects they only run for a few minutes (I know we have some massive outliers like Mesa though)(

16:58 <bentiss> alanc: but that bug shouldn't affect the host, only the container, and you can not directly ssh into a container AFAICT

16:58 <bentiss> (because we already use port 22 on the host, so the container can not bind to it)

16:59 <alanc> oh right, hadn't thought about the port already being bound

16:59 <bentiss> \o/ the db migration now seems to be going further after the restart

17:01 <bentiss> \o/ \o/ migration almost over

17:01 <bentiss> \o/ \o/ \o/ migration done, now we need for webservice to kick in before we enter the era of gitlab 16.10

17:02 <bentiss> alanc: also the containers are using a NAT behind the host, I don't think we use the host network

17:03 <bentiss> and there we go. Gitlab 16.10.1 deployed (finally)

17:07 <alanc> and I know all the CI containers I'd worked on don't have sshd, but I thought I'd seen some way to set it up so you could login to debug stuff, but perhaps I misremembered that

17:08 <bentiss> well, there was a remote service which was doing reverse ssh tunneling that I disabled by redirecting it to 127.0.0.1, but otherwise you probably need some host configuration we don't have

17:20 <__tim> bentiss, nice, seems snappier too, but maybe that's just the public holiday

17:20 <bentiss> __tim: could also be the db that was stuck in limbos, because the migration was hitting some locks that couldn't be released