daniels changed the topic of #freedesktop to: GitLab is currently down for upgrade; will be a while before it's back || https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
utuik has quit [Remote host closed the connection]
killpid_ has quit [Quit: Quit.]
keypresser86 has quit [Ping timeout: 480 seconds]
columbarius has joined #freedesktop
co1umbarius has quit [Ping timeout: 480 seconds]
alpernebbi has quit [Ping timeout: 480 seconds]
alpernebbi has joined #freedesktop
anon_ has joined #freedesktop
ikihiki has joined #freedesktop
ikihiki has left #freedesktop [#freedesktop]
AbleBacon has quit [Read error: Connection reset by peer]
<anon_>
Hi, are there any resources to help me do my own guessing on when the GitLab upgrade will be done? like Is there a post somewhere as to when it started or when it usually starts?
<whot>
anon_: let's say CEST 09:00 plus a few h, that's when it should be done. big db migration and the admin is getting well-deserved sleep while that's running
<anon_>
thanks!
ximion has quit [Quit: Detached from the Matrix]
anon_ has quit [Quit: Page closed]
ids1024 has joined #freedesktop
callum_ has joined #freedesktop
callum_ has quit []
bmodem has joined #freedesktop
dcunit3d_ has joined #freedesktop
dcunit3d has quit [Ping timeout: 480 seconds]
anon_ has joined #freedesktop
psukys has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
bmodem has joined #freedesktop
bmodem has quit [Excess Flood]
bmodem has joined #freedesktop
<bentiss>
morning update: the db restoration went through on both clusters! I have started the restoration of the git data on 3 out of 4 gitaly servers. The last one is currently backuping on the old cluster (and it's the one that's taking 8 hours)
<mupuf>
bentiss: good morning!
<mupuf>
great news :)
<bentiss>
yeah. I wonder if I should not have gitaly-0 (which has mesa) as a hot transfer, because otherwise I'll never enable gitlab today
<mupuf>
bentiss: how do hot transfers work?
<bentiss>
mupuf: it's a gitlab rake command -> you say that you want one repo to be moved from one gitaly server to one other
<bentiss>
but it requires both clusters to communicate properly
<mupuf>
right
<mupuf>
and when the two are in a different datacenter... it is problematic
<bentiss>
it's problematic if submariner which I use to connect the 2 clusters starts failing the connectivity
mvlad has joined #freedesktop
sima has joined #freedesktop
sumits has joined #freedesktop
<Venemo>
any guess on when gitlab will work again?
<mupuf>
Venemo: either later today if the connectivity issue between the 2 clusters can be worked around, otherwise tomorrow
An0num0us has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
colemickens has joined #freedesktop
bmodem has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
happylinks has joined #freedesktop
<happylinks>
there is no read-only mirror for everything somewhere right?
<happylinks>
airlied: thanks for checking, unfortunately I need a very specific branch from a developer that's still sleeping haha, but I'll ask him when he's up!
ballast has quit [Quit: Page closed]
todi has joined #freedesktop
SirDeiu has joined #freedesktop
vjaquez has joined #freedesktop
sravn has joined #freedesktop
<slomo>
yeah that mirror is up-to-date for the branches on the main repo, it's synced from gitlab (and the original location of that repository)
happylinks has quit [Remote host closed the connection]
<viktoria>
oh covered under my remote desktop control icon ^^ sorry
<kolAflash>
SirDeiu: Thanks for the hint about the channel topic :-)
kolAflash has quit [Quit: Page closed]
foo has joined #freedesktop
foo has quit []
<bentiss>
noon-ish update: everything went fine until ~9:30 (2h30 ago) when suddenly the restore of the gitaly backups stalled and ceph was *really* not happy
<bentiss>
I tried to revive it, but I only got the CEPH_OK a few minutes ago
<mupuf>
oh boy
<bentiss>
I suspect submariner (the vxlan that connects both clusters) to mess up with everything as I can not get correct connectivity even after a reboot
<bentiss>
so I disabled submariner, and we'll see if the cluster without it holds the load
gbissett has joined #freedesktop
<bentiss>
meanwhile, I am also restoring the db on an upgraded db on the old cluster. If this goes through (and the new cluster still fails), we can always use this new db and upgrade to gitlab 16.x to get the security updates
<bentiss>
the downside is the old cluster is clearly slower than the new :(
<bentiss>
damn, the new cluster is flickering once more
gx has joined #freedesktop
gx is now known as Guest902
Guest902 has quit []
<suyuan>
Is there any hope of fixing gitlab today?
<bentiss>
suyuan: hopefully yes
<suyuan>
Please~ Come on, thank you for your hard work. looking forward to good news
An0num0us has quit [Ping timeout: 480 seconds]
<mupuf>
bentiss: want me to toot something about the update? I assume we don;t want to tell people to come to #freedesktop to get updates
<bentiss>
mupuf: yes please, but right now I'm explaining where I am but there are a few bits out of my control (why is this new cluster not behaving properlky when it was fine for ~2 months)
<mupuf>
bentiss: ack!
colin has joined #freedesktop
<mupuf>
I'll PM you what I am planning on sending
<bentiss>
mupuf: thanks!
suyuan has quit [Remote host closed the connection]
colin is now known as Guest907
Guest907 has quit []
suja has joined #freedesktop
kxkamil has joined #freedesktop
happylinks has joined #freedesktop
gert31 has quit [Quit: Leaving]
emusia has joined #freedesktop
happylinks has quit [Quit: Page closed]
happylinks has joined #freedesktop
killp has quit []
tprn has joined #freedesktop
tprn has quit []
Ahuj has joined #freedesktop
anon_ has joined #freedesktop
anon_ has quit []
LaughingMan[m] has joined #freedesktop
ascent12 has joined #freedesktop
pac85[m] has joined #freedesktop
tjaalton has joined #freedesktop
happylinks has quit [Remote host closed the connection]
<alyssa>
It's 10am on a Monday and I haven't had any Mesa MRs to review
<alyssa>
This is the dream!
<alyssa>
;P
An0num0us has quit [Ping timeout: 480 seconds]
suyuan has joined #freedesktop
AbleBacon has joined #freedesktop
<slomo>
alyssa: also no new issues reported! all the code is finally bug-free, apparently
Haaninjo has joined #freedesktop
<alyssa>
slomo: Should we do this every monday? :p
<slomo>
alyssa: why limit it to mondays? :)
<slomo>
but yes, i don't really mind much to have a day today to get rid of some of my backlog instead of putting new things on it ;) things will come back when they're ready
<ebassi>
You've heard about 4 days work weeks; how about 4 days work months? ;-)
b9AcE has joined #freedesktop
<b9AcE>
Is there some expected end of the GitLab downtime?
<mupuf>
b9AcE: another 6+ hours
<mupuf>
If all goes well
<b9AcE>
Thank you.
b9AcE has quit [Quit: /disco]
user-8747 has joined #freedesktop
Ahuj has quit [Ping timeout: 480 seconds]
kolAflash has joined #freedesktop
kolAflash has quit [Quit: Page closed]
tzimmermann has quit [Quit: Leaving]
SirDeiu has quit [Quit: quit]
SirDeiu has joined #freedesktop
An0num0us has joined #freedesktop
suyuan has quit [Remote host closed the connection]
An0num0us has quit [Ping timeout: 480 seconds]
Trinitro1X has joined #freedesktop
Trinitro1X has quit []
Inky1003 has joined #freedesktop
<Inky1003>
What really is the Freedesktop GitLab update?
<JimK>
What's the status of the GitLab upgrade? i.e. Is there a better estimate than "a while" before it's back? Hours? Days?
<bilboed>
4 days work months ? I'm sold !
pkira_ has quit [Ping timeout: 480 seconds]
mattst88 has joined #freedesktop
<alyssa>
IDK this is my most productive Monday morning in ages, let the ops team take as long as they need to do it right :~)
<karolherbst>
alyssa: btw, is there a mirror I can fetch from to get like an updated mesa tree for asahi?
<alyssa>
karolherbst: I can dump things to github if you need
<karolherbst>
no worries then, I still have a kernel regression to deal with anyway :') I kinda wanted to do some stuff and then noticed the upstream mesa stuff doesn't work :D
<alyssa>
that's approximately what we're shipping in fedora
Nefsen402 has joined #freedesktop
<karolherbst>
okay, thanks, maybe I get to it again before gitlab got revived
<bentiss>
alright... I'd say the new db has been properly restored, I'll try to re-enable gitlab, please do not push any valuable work as I might need to revert
suyuan has joined #freedesktop
<suyuan>
God. . . gitlab still cannot be opened. 504 time-out I don't want to wait anymore. Maybe there is no time.
<MrCooper>
suyuan: the person doing the work just gave a short status update 5 minutes ago
<suyuan>
Sorry, I didn't see it, please send me an update on the progress. Thank you so much
<mattst88>
> alright... I'd say the new db has been properly restored, I'll try to re-enable gitlab, please do not push any valuable work as I might need to revert
<Nefsen402>
its up for me. It's time to push irreplacable code without backups!
<suyuan>
Thank you very much for your hard work. I just want to read pull down a library. very very very thank you~
<alyssa>
no new MRs to review, amazing! today is a great day to a be a mesa maintainer :^)
<bentiss>
registry seems to be down
<bentiss>
(which means no CI)
<MrCooper>
alyssa: enjoy it while it lasts, the floodgates are opening again :)
<alyssa>
nooo
colemickens has quit [Quit: Connection closed for inactivity]
<karolherbst>
no CI? merge all the things before they fail
<koike>
lol
<koike>
unique opportunity
<mupuf>
bentiss: the registry seem to accept container pushes, at least
alyssa has quit [Quit: alyssa]
JimK has quit [Quit: Page closed]
dayanf87[m] has joined #freedesktop
suyuan has quit [Remote host closed the connection]
<bentiss>
I think we are relatively in good shape now. I'll go afk until tomorrow to rest a little
<MrCooper>
well deserved! Thanks for your hard work
<mupuf>
bentiss: thank you, Benj!
<mupuf>
Enjoy your evenin
<mupuf>
g
spiegela has quit [Ping timeout: 480 seconds]
siqueira_ is now known as siqueira
<philn>
thanks bentiss!
<mupuf>
the registry seems to be collapsing. Probably due to too many requests from CI
<mupuf>
it answers.... but it is super slow
V has quit [Ping timeout: 480 seconds]
ximion has joined #freedesktop
killpid_ has quit [Quit: Quit.]
<dabrain34[m]1>
Thanks bentiss
kxkamil has quit []
slattann has quit [Ping timeout: 480 seconds]
kxkamil has joined #freedesktop
<dabrain34[m]1>
yes I get manifest unknown so it triggers a new docker build