09:15 <sergi> eric_engestrom: Now, with !34120 merged and the farms with nginx-proxy/cache problem disabled, we can address that calm but committed. None of my experiments to understand the problem gave me any useful information. How can we sync to find a root cause affecting those proxies?

10:22 <eric_engestrom> sergi: a workaround has been found by jasuarez, MR coming soon :)

10:25 <sergi> great! thanks!

10:26 <eric_engestrom> sergi, daniels: please send this to anyone with a lava or baremetal farm: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34174

10:26 <eric_engestrom> the igalia farm already has that change and is about to be re-enabled

10:34 <eric_engestrom> (igalia farm re-enablement: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34175)

10:55 <daniels> enunes: ^ for Lima

10:55 <daniels> blu: ^ for vmw

10:55 <daniels> robclark: ^ for fdno

11:10 <eric_engestrom> thanks!

11:14 <enunes> daniels: eric_engestrom: I had the old snippet indeed, applied the the fix now

11:36 <blu> heya, so I've patched my linux branch to upload artifacts via curl now, and it seems to upload fine, but for some reason I cannot consume any of the new artifacts under https://s3.freedesktop.org/mesa-lava/.. -- the files are just 404. There used to be a better folder for these artifacts, right?

11:51 <valentine> blu: Hey, I checked your branch, and the problem is that your S3 URL contains the filename again

11:52 <valentine> with the recent changes, the upload URL should simply be "https://${S3_PATH}/"

11:53 <valentine> See https://gitlab.freedesktop.org/gfx-ci/linux/-/commit/c430eb0106833e41ade7bcb5018b806afc7dac85 with our fix

11:53 <blu> valentine: oops. thanks!

11:54 <valentine> No problem :)

12:16 <daniels> blu: please please rebase your kernel so it goes to mesa-rootfs and not mesa-lava

12:16 <daniels> mesa-lava expires after a month; mesa-rootfs does not

12:16 <daniels> so if you do this, we don't have to worry about the vmware jobs all failing because someone needs to regenerate the kernel again

12:16 <blu> daniels: ah, that's the folder I was looking for. thank for that too!

12:17 <daniels> (using the kernel from gfx-ci/linux would also be really great)

12:25 <robclark> eric_engestrom: that is a change I need to make locally to the farm?

14:11 <__tim> do we need to bump the ci-templates commit in our ci pipelines for the image jobs to work again or will that fix itself once the remaining registry stuff is sorted?

14:13 <__tim> (current fails with 'operation not permitted' doing 'podman login', but mesa pipelines seem to be working)

14:18 <bentiss> __tim: job link?

14:18 <__tim> e.g. https://gitlab.freedesktop.org/xhaakon/gstreamer/-/jobs/73268277

14:19 <bentiss> placeholder-job are not privilged

14:19 <__tim> ah, so need to add the kvm label?

14:19 <bentiss> remove the placeholder label basically

14:20 <__tim> hrm ok

14:20 <bentiss> I'm seriously considering how to solve that "placeholder" problem

14:21 <bentiss> it used to be "let's keep some long standing jobs around without limitting our capacity", but now it's used as "fast forward the queue and have the job running now"

14:21 <__tim> it was handy imho that these image jobs which 99.999% of the times do nothing and finish within seconds had priority, because they block the rest of the pipeline

14:21 <bentiss> it's a valid use case, but probably the implementation is wrong

14:21 <__tim> ok

14:22 <bentiss> __tim: FWIW, slapping kvm should give you a fast forward too ATM, but not in the long run

14:23 <__tim> would you recommend we do that for now, or should we just wait until some other solution is found later?

14:24 <__tim> we don't want to game/abuse the system either of course

14:25 <bentiss> yeah, that should be fine. I don't have a full solution for this. We briefly talked about this at plumbers with daniels, but right now I'm not sure I'll have the bandwidth wo draw a new solution :/

14:26 <__tim> alright, I'll add 'kvm' for now then, and please shout if it causes problems

14:26 <bentiss> __tim: thanks :)

14:27 <__tim> Thank *you*!

14:31 <blu> robclark: that change needs to go into the nginx config of your farm. please see set_by_lua here: https://docs.mesa3d.org/ci/bare-metal.html#caching-downloads

14:34 <robclark> ok, yeah, I eventually figured that out.. the next challenge is that I don't actually have sudo on the box ;-)

14:34 <blu> robclark: oh :/

14:43 <robclark> ok, I think I got that sorted

14:45 <__tim> ooc, is runner capacity already at the full planned level, because I'm seeing things being queued 25-35 minutes in the mesa pipelines

14:46 <Ford_Prefect> PipeWire pipelines are also crawling rn

14:48 <__tim> ah, I guess I should have re-read the ticket