06:49 <alatiera> daniels there still 500s it seems, and when trying to click "run pipeline" to trigger a new one it returns an internal error

06:49 <alatiera> https://gitlab.freedesktop.org/ferjmoreno/gst-docs/-/jobs/10285879

06:50 <alatiera> > An error occurred while trying to run a new pipeline for this merge request.

07:14 <bentiss> alatiera: 500s are still expected, we are still on the same wonky server for artifacts. The move is undergoing

07:19 <alatiera> oh ack!

07:19 <alatiera> Daniel had mentioned something yday about being over so thought it was a new or recurring issue popping up again

07:21 <bentiss> alatiera: well, some times it seems over, but right when we say it, 500s pop up, so we ended up decinding to switch to an other system, but migrating data takes time

07:22 <bentiss> alatiera: well, some times it seems over, but right when we say it, 500s pop up, so we ended up decinding to switch to an other system, but migrating data takes time

07:22 <bentiss> (talking about ~1000000 files over 2.2 TB)

07:42 <bentiss> daniels: I have started again `mc mirror --watch minio-artifacts test-ceph` on server-2. Once this has settled I'll switch the artifacts server in the configs

07:42 <bentiss> --watch is important or we'll lose the current jobs

09:30 <daniels> bentiss: ah, thankyou

10:00 <bentiss> daniels: FWIW, I messed something up and I can not enable the ceph dashboard on the new cluster

10:00 <bentiss> Unhandled exception from module 'dashboard' while running on mgr.a: OSError("No socket could be created -- (('10.41.168.156', 8445): [Errno 99] Cannot assign requested address)",)

10:01 <bentiss> it tries to bind on the service IP, which a pod can not do...

10:01 <bentiss> not sure what happened

10:02 <bentiss> I mean, I know what I did (drained large-4), but I don't understand why the mgr-a pod does that now when it wasn't the case previously

10:12 <MrCooper> any idea why gitlab doesn't let me enable/disable notifications on wayland-protocols MRs? Specifically, I'd like to enable notifications for https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/67, but the switch is greyed out

10:12 <gitlab-bot> wayland issue (Merge request) 67 in wayland-protocols "unstable/drm-lease: DRM lease protocol support" [Needs Acks, Needs Implementations, Needs Review, New Protocol, Opened]

10:13 <daniels> bentiss: yeah, that's weird since the dashboard was working last night ...

10:14 <bentiss> it was working this morning, but as soon as I deleted the pod on large-4, it started failing

10:28 <pq> MrCooper, I wondered about the same issue in kwinft repo which is in gitlab.com.

12:00 <bentiss> daniels: the issue with the dashboard is fixed now -> https://github.com/rook/rook/issues/7854 I had to run in the ceph-toolbox pod ceph config set mgr.a mgr/dashboard/server_addr 0.0.0.0

12:00 <gitlab-bot> rook issue 7854 in rook "ceph mgr attempts to bind to the wrong pod IP for prometheus+dashboard" [Bug, Open]

15:06 <bengal> hi, the captcha on the gitlab new user signup page is broken due to CSP. Is that a known issue?

16:37 <yk> Has there been some change to GitLab's notification system recently? I can't seem to subscribe/unsubscribe to individual issues or MRs anymore.

16:44 <yk> It seemes to be tied to the role. My colleague, who's a Reporter, can subscribe to issues, but not MRs.

16:47 <imirkin> yk: what's your role? (i have no idea what the issue is, but the answer to that q may be relevant to someone who is more knowledgeable)

16:50 <yk> default role

16:50 <yk> I'm @yk at gitlab

16:54 <yk> I definitely used to be able to subscribe/unsubscribe to MRs and issues until last weekend.

16:55 <imirkin> daniels: --^

16:55 <imirkin> there may have been a gitlab upgrade, dunno

16:55 <daniels> yep, there was, to 13.12

16:58 <daniels> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61953

16:58 <gitlab-bot> GitLab.org issue (Merge request) 61953 in gitlab "Fix ability for non project member to subscribe to an issue" [Bug, Devops::Plan, Frontend, Group::Project Management, Priority::2, Section::Dev, Severity::2, Workflow::Production, Merged]

17:13 <yk> Oh, got it. Thanks!

19:31 <bentiss> daniels: damn, server-2 is out of memory

19:31 <bentiss> and it starts killing processes everywhere, meaning k3s is not happy

19:36 <bentiss> the sad part is I wanted to re-use server-1 to the new clsuter, so we are effectively not having k3s working in HA right now...

19:37 <bentiss> got to hard reboot it :(

20:06 <bentiss> OK, server-2 is back, I had to lock the kilo container version :(

21:23 <anholt> are we still expecting to see 500s uploading artifacts?