04:48 <imirkin_> looks like gitlab is taking another break

04:48 <imirkin_> getting 500's in the UI, git is not responding

04:49 <imirkin_> cgit is up though. that's at least something :)

04:50 <imirkin_> ... and it's back

06:22 <tomeu> yeah, and could there be a problem with some packet runners? https://gitlab.freedesktop.org/gfx-ci/mesa-performance-tracking/-/jobs/13149799

08:47 <daniels> tomeu: if that job ran around 5:45am UK, it probably worked fine, but the runner wasn't able to post results

08:53 <kusma> Seems gitlab is dead again...

09:03 <kusma> came back up, but is now dead again, it seems...

09:07 <daniels> bentiss: ^ fwiw when I looked into this last weekend, I was seeing webservice pods die because they were unable to connect over TCP to the psql svc, but no errors or restarts logged in the psql pod at all? I'm not sure how to track down if it's a network fabric thing or psql itself?

12:04 <shadeslayer> I've hit a weird bug where opening failures.csv in https://gitlab.freedesktop.org/mesa/mesa/-/jobs/13158813/artifacts/browse/results/ leads to a Error 500 message

12:04 <shadeslayer> The file does exist but is empty, I'm wondering if this is a bug in gitlab or in the fdo instance?

13:34 <bilboed> more 500's from gitlab btw

13:38 <tomeu> shadeslayer: was it maybe a transient error?

14:02 <daniels> yeah, 500 + 502 is transient

14:35 <alatiera> what's our gitlab version?

14:57 <daniels> 14.1

15:04 <alatiera> thanks

15:16 <daniels> service is repaired

15:17 <daniels> bentiss: we should probably enable persistent journal; I couldn't see why k3s-server-4 was so dead and unresponsive, but if it's anything like server-2 the other day, it was just buried in D-state procs and timing-out page flushes