08:18 <mupuf> DavidHeidelberg[m], daniels: Seems like acer-R721T-grunt-cbg-2 is having a very bad day

08:18 <mupuf> https://gitlab.freedesktop.org/mesa/mesa/-/jobs/33451146 and https://gitlab.freedesktop.org/mesa/mesa/-/jobs/33422436 both failed because depthcharge failed

08:24 <mupuf> I filed https://gitlab.freedesktop.org/mesa/mesa/-/issues/7893

08:27 <daniels> mupuf: thanks, LAVA already picked that up earlier this morning and jobs aren't being scheduled to it until it's fixed

08:29 <mupuf> daniels: good, I have a feature like this on my todo list (machines that fail to get to the point where they would download the kernel/initramfs a couple of times get automatically kicked out)

08:30 <mupuf> now, on to this flake: https://mesa.pages.freedesktop.org/-/mesa/-/jobs/33476751/artifacts/results/summary/results/trace@freedreno-a630@unvanquished@unvanquished-lowest.trace.html

08:33 <daniels> enjoy!

08:47 <bentiss> daniels: one thing I realized we could easily do to cut the GKE costs again would be to deploy an internal quay and also push the mesa/valve-infra/gstreamer images there too

08:48 <bentiss> daniels: because quay has 2 things we want (since 3.8, out last week): quota usage, so we can limit how many images we keep, and transparent cache for other images if we want

08:50 <bentiss> daniels: also we can probably ditch the 2.5TB of old artifacts/uploads/lfs/packages

08:51 <bentiss> problem is, I really don't have time to deploy quay right now :(

09:29 <DragoonAethis> bentiss: Harbor does both of these things too and is pretty easy to deploy (at least as far as I needed it to go)

09:29 <DragoonAethis> Is it missing anything you need?

09:31 <bentiss> DragoonAethis: not necessary anything special. AFAICT, the quota usage on quay is smart: you can still upload new images, and it'll automatically remove the less used ones, which is what we want

09:31 <bentiss> don't know much about harbor TBH

09:32 <bentiss> hmm... harbor has a native helm chart, which is what we use to deploy everything. When quay is relying on operator

09:33 <DragoonAethis> I've looked at the quota demo in Quay and it only had policies for warn/reject, not deleting the old ones

09:33 <DragoonAethis> Harbor can only reject, too

09:35 <bentiss> DragoonAethis: looks like quay 3.8 is better than it was. I saw a demo of the transparent cache, and it was using quota in that interesting way

09:37 <DragoonAethis> https://access.redhat.com/documentation/en-us/red_hat_quay/3.8/html/manage_red_hat_quay/red-hat-quay-quota-management-and-enforcement - doesn't seem to look like it?

09:38 <DragoonAethis> Maybe it's just for the caching proxy (Harbor also has automatic retention policies for proxied objects for 7 days/least recently used on quota exceeded, but not for hosted repos)

09:45 <daniels> bentiss: hmmm, so you mean to replace the existing registry proxy with quay? or to have a third registry?

09:51 <bentiss> daniels: yeah, replace the current proxy but also make it the main one for mesa/pipewire/gstreamer

09:52 <bentiss> basically we can rewrite the registry-mirror.fd.o to quay.fd.o/cache and have a quay.fd.o/mesa

09:53 <bentiss> to ensure we keep an archive we can push to both GKE and quay so that if we prune old images, we still have them on GKE

09:53 <bentiss> (replace quay with harbor if that is the choice in the end)

09:54 <mupuf> bentiss: Ah, finally a registry that handles removing images without having to restart?

09:56 <bentiss> mupuf: technically the new gitlab registry is capable of doing that, but we are still waiting for the migration to be marked as "production ready" by gitlab (and not sure they will ever do, because I remember having read that they consider self hosted deployments can just migrate offline)

09:57 <daniels> bentiss: ooh right - I was meaning over Christmas to check how the new registry migration was going ...

09:58 <bentiss> daniels: FWYI, given that I said I don't have time to work on it, I already manually deployed https://github.com/operator-framework/operator-lifecycle-manager to be able to use operator-capable things on our cluster :)

09:58 <mupuf> bentiss: I see... that's indeed unfortunate. Do you know if there is an issue for it on gitlab? If not, I may write one

09:59 <bentiss> mupuf: not sure they have one specific for self hosted. Though they already did the migrtion on gitlab.com, so everything should be ready, minus a little bit of tinkering

10:00 <mupuf> I see :)

10:00 <mupuf> Thanks for all your hard work!

10:00 <bentiss> can't remember the state of the chart deployment if we can specify the registry db or not (I vaguely remember we could)

10:01 <bentiss> but the more I think of it, the more I'd prefer first having an external registry we can manage with the gitlab's one being backup or for forks

10:01 <bentiss> for the main projects that are using the bulk of the bandwidth I mean

10:01 <daniels> bentiss: oh nice, that's cool, thanks :)

10:02 <bentiss> daniels: litterally just running https://github.com/operator-framework/operator-lifecycle-manager/blob/master/scripts/install.sh locally :)

10:02 <daniels> heh

10:02 <bentiss> AFAICT, the operators should handle the automatic update of the components in our back

10:03 <bentiss> so if we can have quay update itself for security reasons, that would be nice I think

12:31 <mupuf> can anyone review this change: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20312 ?

12:31 <mupuf> it is addressing a flake

13:22 <bentiss> DragoonAethis: looks like the quay-operator is too much tighten to openshift, and it fails at fetching the route API, which is openshift native

13:22 <bentiss> so I guess it'll probably be harbor

14:24 <DragoonAethis> bentiss: it's almost pleasant to work with, enjoy :D

16:40 <mupuf> DragoonAethis: ROFL

19:07 <bentiss> daniels, DragoonAethis: so I managed to deploy harbor, configure it as a public proxy and just tested gc, it all works ;)

19:08 <DragoonAethis> bentiss: Excellent

19:08 <bentiss> now we need to find dedicated volunteers to convert ci-templates and mesa

19:08 <bentiss> daniels: I'll push the changes later tonight

23:51 <__tim> what kind of machine do the placeholder-equinox jobs run on? It looks like they don't cache docker images locally or something? I've been seeing it pull the same image every time (taking 4 minutes before running the little script they're supposed to run)