daniels changed the topic of #freedesktop to: GitLab is currently down for upgrade; will be a while before it's back || https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
columbarius has joined #freedesktop
co1umbarius has quit [Ping timeout: 480 seconds]
konstantin_ has joined #freedesktop
konstantin has quit [Ping timeout: 480 seconds]
AbleBacon has quit [Read error: Connection reset by peer]
<karolherbst>
ehh.. also need to update other places..
<karolherbst>
yeah.. well.. it fails 5 out of 5 pipelines I'm aware of running those jobs in the last 10 hours, and I think we really want to land that crosvm fix to unblock other MRs where people already got annoyed by :')
<karolherbst>
anyway.. retried that one as it was kinda stuck for 15 minutes not doing anything
<karolherbst>
"wget: bad address 'gitlab.freedesktop.org'" :')
<karolherbst>
but that retry magic is really cool honestly
aninternettroll has joined #freedesktop
Ahuj has joined #freedesktop
<mupuf>
bentiss: minio.error.S3Error: S3 operation failed; code: XMinioStorageFull, message: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed.
<mupuf>
Oh, sorry, that is likely on my end!
Major_Biscuit has joined #freedesktop
Major_Biscuit has quit []
i-garrison has quit [Remote host closed the connection]
i-garrison has joined #freedesktop
<bentiss>
mupuf: yeah, we don't have minio anymore on the gitlab side
<mupuf>
yeah, it was on my gateway
<mupuf>
sorry about that
<mupuf>
it's fixed now
<mupuf>
half the drive was used by old volumes, containers, and caching of the fdo containers
<bentiss>
and FWIW, no 500 error on the registry over the night :) \o/
<karolherbst>
noice
mirai_ has joined #freedesktop
mirai has quit [Remote host closed the connection]
An0num0us has joined #freedesktop
ximion has quit [Quit: Detached from the Matrix]
tzimmermann has joined #freedesktop
<mupuf>
bentiss: that's amazing!
<mupuf>
should increase CI's reliability nicely :)(
<bentiss>
changing the data backend of the registry now so we disconnects more from the failing cluster
bmodem has joined #freedesktop
nedko has quit [Remote host closed the connection]
<bentiss>
looks like something is not happy there
<bentiss>
got a lot of blob unknown to registry
<bentiss>
could very well be that the ones I tried were failing already
MajorBiscuit has joined #freedesktop
<bentiss>
switching back to the previous data storage
konstantin_ is now known as konstantin
<bentiss>
doesn't seem to change a bit, so returning to the main cluster
mirai_ has quit [Remote host closed the connection]
mirai_ has joined #freedesktop
<mupuf>
bentiss: yeah, I found a lot of missing blobs, even in ci-templates
<bentiss>
oops, I think I found out why: there was a missing config in the new registry deployment in which it told users to directly fetch from S3. Not sure how this could have been working yesterday
<hch12907>
hi, I just noticed gitlab.fd.o doesn't seem to have ipv6 connectivity. is it because of the migration?
<bentiss>
hch12907: it could have if the migration happened properly, but the new cluster is not happy, so nopt
<bentiss>
nope
<hch12907>
i see
<hch12907>
but still, the migration must've been stressful, keep up the good work guys
<mupuf>
bentiss: is there an http frontend that could be caching requests to the registry?
<bentiss>
mupuf: nope. I don't have redis enabled for the registry
<mupuf>
ack
<mupuf>
I just tried to run skopeo in a container, either fedora or arch, and I keep getting the same result
<mupuf>
bentiss: I asked eric to inspect registry.freedesktop.org/freedesktop/ci-templates/container-build-base:2023-07-12.1 and he also gets "blob unknown to the registry"
<mupuf>
I'll try to repush it myself
<bentiss>
mupuf: can you please retry now?
<mupuf>
same
<bentiss>
yeah, maybe retry pushing it
mirai_ has quit [Ping timeout: 480 seconds]
<mupuf>
bentiss: done, and it works
<mupuf>
and the runners seem happy now
<bentiss>
strange, but thanks :)
<mupuf>
indeed
hch12907 has left #freedesktop [#freedesktop]
<mupuf>
bentiss: the images i built this morning were pushed to gcs, not the registry
<mupuf>
is that related to packet being configured to push to gcs?
<mupuf>
I thought it would only affect the fdo runners. In my case, I pushed it from my runners
bmodem has quit [Ping timeout: 480 seconds]
vkareh has joined #freedesktop
mripard has quit []
<bentiss>
mupuf: it was the registry config. It was pointing at gcs, so whatever you pushed and from anywhere, it was going to gcs
<DavidHeidelberg[m]>
I would blame it on the infra I guess, it never took that much, probably overloaded a lot, if you have two samples, disabling would make sense
emery has joined #freedesktop
<zmike>
I've had this happen multiple times over the past couple months
viktoria has quit [Quit: Page closed]
AbleBacon has joined #freedesktop
Ahuj has quit [Ping timeout: 480 seconds]
<pq>
Is there still an elevated risk that review etc. comments posted on gitlab might be lost?
mripard has joined #freedesktop
<mupuf>
pq: shouldn't be an issue
<pq>
cool!
<bentiss>
heh, I'm playing with a dump of the registry db, and it seems that noone but ci-templates is using the "fdo.expires-after" label
<bentiss>
still, that's 5173 registry images to prune :)
Major_Biscuit has joined #freedesktop
<bentiss>
my bad, gst-editing-services, gst-examples and a few others are using it :)
<bentiss>
now I wonder if I should mass delete them or not
MajorBiscuit has quit [Ping timeout: 480 seconds]
vyivel has quit [Read error: Connection reset by peer]
vyivel has joined #freedesktop
mvlad has joined #freedesktop
Haaninjo has joined #freedesktop
<DavidHeidelberg[m]>
zmike: hmm, then I guess we'll do separation, so feel free to kill it for now. I'm making a TODO to split it into separate job.
<zmike>
what exactly am I killing
<DavidHeidelberg[m]>
zmike: the nouveau part of shader-db test, if you noticing it takes unsual amount time sometimes
<DavidHeidelberg[m]>
then we split shader-db away and set reasonable timeout for it