09:19 <bentiss> thaller: in addition to what whot said: in theory, I have switched all upstream projects to use teh "new" gitlab registry which supports online garbage collection. So if you remove the tags from the web UI (or overwrite the existing ones), the blobs should be cleared.

09:20 <thaller> bentiss, great. thanks. that solves my concerns (about wasting resources due to unused images)

09:20 <bentiss> thaller: but we are still in a transistion period, so it's not 100% sure everything will work. We are still running both old and new registry at the same time, so some weird interactions happens

09:21 <thaller> bentiss, it's fine, if they "eventually" get cleaned up.

09:22 <bentiss> thaller: great then!

09:25 <thaller> bentiss, to us, those cached images are only for performance (to safe bandwidth+runnertime+cputime resources). As such, they can always be dropped and regenerated (even if the regenerated image is not exactly the same)

09:26 <thaller> (I guess, to some projects those images might be important to exactly reproduce something later? We don't look back for gitlab-ci-results, and only care for the latest test).

09:40 <bentiss> thaller: I guess most projects care about the latest images only. It's useful for debugging what went wrong, but in 99% of the cases, projects are merging when the pipeline succeed, so there is no reasons to do forensics on 2 months old images

09:47 <bentiss> daniels, mupuf: so I was playing last week with Fedora CoreOS (locally and on equinix metal). Turns out that I should be able to have the proper scripts to deploy this automatic updating distro. I wonder if I should not replace arm-8 with a test CoreOS, given that the x86_64 runners are fine while we need to take care of the arm ones every 3 weeks

09:48 <mupuf> bentiss: oh, I'm all for immutable distros

09:49 <bentiss> daniels: also, I was thinking that given that we are going to take the cluster down for the gitlab 16.0 update, *maybe* we could use that opportunity to: 1. change data center, and regroup with the runners (which I believe equinix would prefer us), 2. rebuild a new cluster there with an updated k3s AND IPv6 support

09:50 <bentiss> The reasons is that we need to upgrade postgres, but we also need to change the k3s network backend. So maybe it's a good opportunity to do that

09:50 <bentiss> (and 3. finalise the migration of the registry to self-hosted data to reduce the GCP costs from $1400 to $400 a month)

09:52 <bentiss> the only problem, is that the ceph migration would likely need to be done through ceph object multisite, and it's an unkonwn (I really don't want to manually migrate the ceph object data by hand once again)

09:53 <bentiss> we would also need to update the DNS entries, but the "we now have IPv6 support" should be a good enough reason, no?

10:14 <mupuf> bentiss: oh boy, this migration is going to be quite something :o

17:03 <eric_engestrom> this might be a question for here (cc mupuf who I asked elsewhere): why do we have a giant list in `.gitlab-ci/common/generate-env.sh` that we have to maintain, instead of dumping everything with `env`? I'm guessing it might be for reproducibility, ie. we want to only pass specific envvars and not anything else that might also be set some other way than gitlab-ci.yml?

17:12 <anholt_> sounds like a possibly nice cleanup

17:12 <anholt_> we have a giant list because it started out as a short list.

17:33 <eric_engestrom> ack, I'll send an MR tomorrow then and see what the reactions are :)

20:30 <Venemo> can I propose a new message for marge bot? Instead of "I couldn't merge this branch: CI failed!" -> "I don't wanna merge this branch because I'm too lazy to run the tests"