06:54 <bentiss> DavidHeidelberg: yeah, s3 is fine. The only question is how to get your file there. If you can upload it through a job, then that's easy enough. If you need to manually upload it, we might need to create a special bucket for manual uploads. But we'll have to get a JWT from gitlab for that...

07:22 <mupuf> between S3 and the container registry?

07:22 <mupuf> bentiss: I would like to add a new test container in mesa, for arm64 testing. This would be used by both imagination and freedreno. Looking at the current code, it seems like we are storing the arm64 rootfs in two places: S3 (for lava), and in a container (bundled with other stuff and extra dependencies) in baremetal. Ideally, I feel like we should have one container that contains all the userspace stuff (stored as zstd:chunked to reduce

07:22 <mupuf> bandwidth when making new versions) which could be used directly by gitlab runners / CI-tron, then have the lava/baremetal jobs create the rootfs they need at run time by extracting the container to NFS and downloading/extracting the kernel they want to use in the same way they currently download them just like they currently do, except by using skopeo/podman rather than wget. Do you have any thoughts on this? Is storage/bandwidth cost the same

07:39 <bentiss> mupuf: no real thoughts on this. storage/bandwidth cost the same between s3 and container registry as it's the same backend (one is serving files directly, the other has a container registry in front of an internal s3)

07:39 <mupuf> good, this matches my expectations

07:40 <bentiss> it just feels more work to pull the container, extract it, when using direct s3 can be cached by a local proxy

07:40 <mupuf> the container can be cached by a local proxy just as well, no?

07:40 <bentiss> yeah but you need to extract it everytime

07:41 <bentiss> from an admin pov, I have more visibility on the s3 filesystem when used directly as the paths are encoded directly (no hash checksum in the paths)

07:41 <bentiss> so if we get out of storage, I can more easily pinpoint who is the responsible

07:41 <mupuf> oh, actually, this is what happens with the current way (the rootfs is download and extracted every time) whereas the container would be extracted on the first download and just copied to NFS

07:42 <bentiss> when using the registry, between everybody copying the image, we have more chances of having stalled intermediate images and no ideas if we can delete those

07:43 <mupuf> The container way should be faster than the current solution, on top of reducing bandwidth needs when new versions get download thanks to zstd:chuncked

07:43 <mupuf> oh, right, accountability is indeed an issue

07:43 <bentiss> long story short: I won't prevent you to have one common container for everything if that's simpler

07:44 <mupuf> (not that we don't already have the problem with all the other containers we build for mesa)

07:44 <bentiss> for acountability, we should have a script that goes on the entire container registry, look at the labels and purge the out of date images

07:45 <bentiss> mupuf: I'm not saying we don't have the problem, just that this part is more easy to be seen

07:45 <mupuf> yeah, got you :)

07:46 <mupuf> alright, so containers win for local reproducibility of the CI environment and bandwidth reduction but they lose on the ease of accountability of storage

07:46 <bentiss> sounds like a good summary, yeah

07:47 <mupuf> Ack, got it!

07:47 <mupuf> thanks a lot :)

07:47 <bentiss> no worries (not sure I helped much TBH)

07:59 <mupuf> bentiss: You confirmed that both end up hitting the same place, and you shared your accounting concerns

07:59 <mupuf> that's all I needed

08:17 <daniels> mupuf: well, if you rewrote LAVA and bare-metal to consume containers rather than a tarball URL, then fine

08:17 <daniels> but short of that, we'd have to have something which would pull the container, tar it up, store it somewhere and access it, which is ... exactly what we do now?

08:17 <daniels> so I'm not really sure what problem that's solving atm

08:26 <mupuf> daniels: yeah, to really reduce duplication, we would have to rewrite both

08:27 <mupuf> What this would solve is imagination and freedreno being able to run tests on aRM64

08:28 <mupuf> We consume containers replicating what baremetal does it just making the problem worse

08:29 <mupuf> Potman has a rootfs mode, but using it would be very inefficient to use

08:29 <daniels> img and fdno are already running tests on arm64 though?

08:29 <daniels> or do you mean enabling those on ci-tron

08:29 <mupuf> Yeah, new jobs using ci tron

08:31 <daniels> yeah I mean, short of rewriting LAVA/bare-metal to both look exactly like ci-tron does (or completely breaking all the existing jobs, or making them much slower than they already are), the best thing for now is just to replicate what all the other jobs do

08:32 <daniels> we'll never get to the point where DUTs are directly consuming containers because most of them are way too slow (either CPU, I/O, or both - plus limited RAM) to take the hit of unpacking a container image during startup

08:33 <daniels> there could be some kind of local cache which takes a container image, flattens it out to a single tarball, and uploads it some kind of storage service which can be used by the ultimate NFS host, but again that's already what we have

08:34 <daniels> (we're not hitting s3 every time because that would obviously be insane - we have multi-tiered local caching proxies and monitoring on hit rates etc - so s3 is primarily just there for better remote visibility)

08:37 <mupuf> I'm confused because what you guys are doing is just a cruder version of a container runtime, and we could replace that to do the exact same thing without changing anything.for the.DUTs

08:38 <mupuf> I'll give it a try, maybe I 've missed something

08:39 <daniels> well sure, whatever you need to do to provide LAVA a URL to a single flat tarball which contains the rootfs

08:39 <daniels> I'm not really sure what the difference would be with what you're proposing but happy to review MRs

08:40 <daniels> (there's no container runtime on the DUTs obviously, they just boot a single rootfs over NFS)

08:40 <mupuf> Yep

08:40 <mupuf> Lava is the one extracting the rooffs, right?

08:41 <mupuf> Makes sense for its design

08:42 <mupuf> So the best we could do is change baremetal not to package the rootfs in its image and instead download it from the registry

08:42 <mupuf> So the rootfs job would push both a container AND a container

08:42 <daniels> that sounds like a nice cleanup, yeah

08:43 <mupuf> I looked into creating a custom manifest that would point to the same layer... but it doesn't seem possible

08:43 <mupuf> Maybe we could get an http link to the layer for lava to consume though

08:43 <mupuf> Will investigate

08:43 <daniels> indeed LAVA extracts the rootfs - the job you submit has a URL to a tarball, then when it's scheduled the job to a given DUT, the local worker attached to that DUT pulls the tarball and unpacks to the NFS root path

08:43 <mupuf> Ack

08:44 <daniels> so yeah, LAVA could be modified to instead unpack a container image to that path instead

08:44 <mupuf> Thanks!

08:44 <daniels> it just hasn't been the low-hanging fruit to date

08:44 <mupuf> Right, and I doubt I'll be hacking on it

08:45 <mupuf> But baremetal seems more ripe for improvements

08:45 <mupuf> Thanks

08:46 <daniels> np

09:13 <martink> good day! would a kind soul land a hand and send a maintenance MR to mergebot's queue: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28737

09:16 <daniels> martink: sure, done

09:17 <martink> daniels: thanks!

09:18 <daniels> np