09:42 <bentiss> daniels: hey, thanks for taking care of that (I'm on vacation in the mountain so somewhat hard to take time to fix this)

09:42 <bentiss> daniels: I'm fine moving the repos off from gitaly-1, but I would create a gitaly-3 first and move everything from -1 to -3, and then delete -1

09:50 <bentiss> hmm... there is a 14.6.0 upgrade too, maybe I'll piggy back this one too

10:16 <bentiss> about to start the gitlab upgrade. There is an nginx update too, so maybe it won't be that simple to do

10:29 <bentiss> that was fast. upgrade to gitlab 14.6.0 and nginx 1.0.4 done

10:38 <bentiss> daniels: FWIW,I had to delete one nginx pod it was stuck in the non running state

10:44 <bentiss> daniels: so all that is needed now is to follow https://gitlab.freedesktop.org/freedesktop/helm-gitlab-config/-/blob/master/misc-gitlab-scripts/move_between_shards.md#even-better-check-all-source-of-forks-from-a-given-shard to empty gitaly-1

11:22 <bentiss> hmm, the nginx pod blocked once again. I hope it's just because of the gitaly-1 situation that blocks all new connections to thepod

11:54 <daniels> great, will do, and thanks for nginx as well - I didn’t get to check what was required for that upgrade yet

11:55 <daniels> I’ll get the migration going in the background; it’s changed a little bit since your steps but hopefully not too much

11:55 <bentiss> daniels: well, I synced up the scripts with the packet-ha branch

11:55 <bentiss> now both master and packet-ha have the correct steps

11:56 <bentiss> I am currently moving NetworkManager forks, there are heaps of them

11:56 <bentiss> (FWIW, it was raining outside, so no reasons to go out :( )

12:02 <daniels> thanks a lot! sorry to hear about the rain

12:03 <daniels> I’m just making coffee then happy to handle the rest if you want to relax

12:09 <bentiss> heh, thanks. Well, I'll see how much I can do

12:09 <bentiss> NetworkManager and its forks is now fully on gitlay-3

12:18 <bentiss> zbus is done too

12:19 <bentiss> FWIW, I am randomly looking at complains from people and chose an order based on that (and on a rough idea of how many forks the base project has)

12:58 <daniels> #<Project id:1353 gstreamer/gst-plugins-good>>, #<Project id:1352 gstreamer/gst-plugins-base>>, #<Project id:891 xkeyboard-config/xkeyboard-config>>, #<Project id:163 libnice/libnice>>, #<Project id:5 drm-hwcomposer/drm-hwcomposer>>, #<Project id:1341 gstreamer/common>>, #<Project id:75 spice/win32/qxl-wddm-dod>>, #<Project id:890 fontconfig/fontconfig>>, #<Project id:10932 ultramarine-linux/anaconda>>, #<Project id:953

12:58 <daniels> pkg-config/pkg-config>>, #<Project id:1357 gstreamer/gstreamer>>, #<Project id:5405 kpouget/presentations>>, #<Project id:2922 daniels/kms-quads>>, #<Project id:4757 pipewire/pipewire-org>>, #<Project id:10606 asahiocean/AnimateLine>>, #<Project id:167 libevdev/libevdev>>, #<Project id:380 spice/x11spice>>, #<Project id:184 mobile-broadband/ModemManager>>, #<Project id:810 pulseaudio/pulseaudio>>, #<Project id:8262

12:58 <daniels> gfx-ci/tracie/traces-db-private>>, #<Project id:1351 gstreamer/gst-plugins-bad>>, #<Project id:1187 dbus/dbus>>

12:58 <daniels> ^ sorted by size of fork network

12:58 <daniels> I'll start with the GSt ones

13:05 <daniels> fwiw that was:

13:05 <daniels> g1 = Project.all().filter { |x| x.repository.shard == "gitaly-1" }

13:05 <daniels> sorted = forks.sort { |x| if x and x.fork_network then x.fork_network.projects.count else 1 end }.reverse

13:05 <daniels> forks = g1.map { |x| if (x.fork_network) then x.fork_network.root_project else x end }.uniq

13:07 <bentiss> cool, thanks. just started libnice/libnice now

13:07 <bentiss> daniels: we will need to also setup the server hooks towards cgit.fd.o (just a reminder)

13:10 <daniels> on gitaly-4?

13:10 <daniels> I'm going to do libevdev + dbus in parallel btw

13:10 <bentiss> no, the server hooks for projects we have mirroring in place towards cgit.fd.o

13:11 <bentiss> yeah doing things in parallel is fine

13:11 <bentiss> (I'm doing one because on a remote laptop)

13:12 <daniels> all good :) if you want to go and look out the window at some wet mountains then go for your life, the only things I was planning to do today were this and making dinner, and being in London is significantly less exciting than the mountains :P

13:12 <daniels> (and right, iswym about server hooks - because when we move the repos we don't move the custom_hooks)

13:13 <bentiss> heh, OK, I'll probably let you play with that very fun project then (feels like a Windows defragmentation, no???)

13:15 <daniels> :D

13:16 <daniels> sounds great, thanks a lot for standing up the new host before I got back

13:16 <bentiss> huh, I am running fstrim on server-2 to free up space on ceph on the gitaly-1 pod and it seems all my connections to the cluster got reset

13:17 <bentiss> anyway I'll check from time to time today. and no worries, it was rainy :)

13:18 <daniels> yeah, I lost all my cluster conns as well

13:18 <daniels> sounds good! have some raclette or something to warm up

13:18 <bentiss> no kidding: raclette was yesterday evening :)

13:19 <daniels> ahhhh, c'est bon

13:29 <ppascher> Thanks to both of you.

13:29 <daniels> np!

13:41 <tanuk> https://gitlab.freedesktop.org/ says "502 Whoops, GitLab is taking too much time to respond. Try refreshing the page, or going back and attempting the action again. Please contact your GitLab administrator if this problem persists."

13:43 <tanuk> It's back now.

13:46 <daniels> tanuk: yeah, we're kind of DoSing it by moving all the repos

13:46 <daniels> (PulseAudio is next)

13:46 <daniels> see the broadcast message at the top of the page

13:47 <tanuk> Ok, I see.

13:57 <daniels> bentiss: btw, I'm thinking to disable persistent storage for Redis and have it just use a host emptyDir, since there's no reason to smash Ceph with all its ephemeral data - I can see it starting to struggle with disk latency when doing bulk moves

13:58 <bentiss> daniels: works for me (I know very little on redis, so I trust you)

14:00 <daniels> mmm, damn, they have some non-cache Redis uses now :\

14:00 <daniels> nm

14:13 <bentiss> daniels: well, then maybe use the same class than the backups: so they will be on a local storage path

14:13 <bentiss> (though it may be non ssd)

14:16 <daniels> bentiss: hostpath isn't persistent though right? i.e. if it gets scheduled on to a different node then it'll lose its data

14:18 <bentiss> daniels: the class of the backups ensures the pod gets always scheduled on the same node

14:18 <bentiss> the controller adds a tag on the pod deployment

14:21 <daniels> right

14:21 <daniels> makes sense

14:22 <daniels> I think it might make sense to partition our services between different hosts anyway, so e.g. Redis/Postgres don't get slammed by Gitaly

14:25 <bentiss> *nods

14:26 <bentiss> (too much muscle memory with kubectl)

14:40 <daniels> haha

16:20 <bentiss> daniels: how is the transfer going on?

16:21 <daniels> bentiss: yeah getting there; one window is moving through all the old pre-merged GSt repos, and the other vacuuming up the smaller fork networks

16:21 <daniels> after those are done I'll shift all the orphans in bulk

16:21 <bentiss> daniels: cool, thanks

16:22 <daniels> np :)

22:07 <daniels> bentiss: so they're basically all done by now, but mwen/linux-rpi failed to move (there's a failed repository_storage_move but it has no error associated with it), and inigomartinez/polkit has a scheduled repository_storage_move which appears to have never quite existed

22:07 <daniels> unless you've got any great ideas, I'll just email them asking them to delete + recreate + repush?

22:48 <bentiss> daniels: I think I managed to solve that in the past, but I won't have time tomorrow to deal with it. So delete/recreate/repush might be the simplest for everybody

22:48 <bentiss> and thanks for moving everything

22:49 <bentiss> hopefully this will go better (my fear is that there is one repo that messes up the entire system, and we just moved the problem elsewhere)

22:54 <daniels> mmm yeah, thankyou