ChanServ changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
agd5f_ has joined #freedesktop
agd5f has quit [Ping timeout: 480 seconds]
co1umbarius has joined #freedesktop
columbarius has quit [Ping timeout: 480 seconds]
AbleBacon has quit [Read error: Connection reset by peer]
<mupuf>
mirai: you need to get verified before you can fork anything
<mupuf>
ha ha, gitlab's clock is 53 years in the future :D
<mupuf>
oh, no, it's my laptop's clock that is :D
agd5f has joined #freedesktop
<mupuf>
System clock synchronized: yes --> I doubt it :D
agd5f_ has quit [Ping timeout: 480 seconds]
psukys has joined #freedesktop
keypresser86_ has quit []
<mupuf>
bentiss: may the force be with you!
<mupuf>
Or whatever one can say to wish luck
<bentiss>
mupuf: thanks!
<bentiss>
and sigh... while I woke up: "but why don't I also tdo the full gitlab db migration while I said that there will be a downtime???"
gchini has quit [Quit: Leaving]
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
todi has quit [Remote host closed the connection]
<mupuf>
Lol
<mupuf>
Well, if the registry goes well, then maybe you'll have time for gitlab too
ximion has quit [Quit: Detached from the Matrix]
sima has joined #freedesktop
<bentiss>
Shutting sown gitlab and gitalb-registry access now, and starting the migrations
<bentiss>
*down
<daniels>
bentiss: bonne chance!
<daniels>
I’ll be around for most of the day if you need but only sporadically as I’m on trains
<bentiss>
daniels: hopefully by tonight we will have gitlab 16.3 :)
<bentiss>
daniels: no worries, let's hope I can cope with everything by myself
todi has joined #freedesktop
<bentiss>
migrating the registry was too easy.... it's done now, though non functional given that gitlab.fd.o is deactivated
realgurka has joined #freedesktop
<mupuf>
bentiss: crossing fingers
lack has quit [Read error: Connection reset by peer]
realgurka has quit [Remote host closed the connection]
<bentiss>
what worries me is that sometimes I lose connectivity between the 2 clusters on some nodes only, and this is going to be a problem for the hot migration of the git data
agd5f_ has joined #freedesktop
agd5f has quit [Ping timeout: 480 seconds]
todi has quit [Ping timeout: 480 seconds]
psychon has joined #freedesktop
todi has joined #freedesktop
Haaninjo has joined #freedesktop
<karolherbst>
"504 Gateway Time-out" :')
<mupuf>
karolherbst: yep, gitlab is being relocated to another datacenter
<karolherbst>
ahh
<karolherbst>
but it's also Sunday and it's my fault for checking emails :D
<bentiss>
TBH, I think I'll also migrate the gitaly data offline, this will be a slightly longer downtime, but it'll save me 2 weeks of monitoring the gitlay transfers.
<mupuf>
bentiss: sounds fair
<mupuf>
hope everything is going smoothly for you
<bentiss>
just to be clear "slightly" in that a backup of gitaly-0 (which I just started and which is the longest) is roughly 8h, so probably same anmount of time to restore
<bentiss>
I could reuse the backups from this morning, but that would drop the changes between 4 to 8 AM UTC. Not sure if there were a lot
<mupuf>
bentiss: so... 16 more hours of work... doesn't seem workable
<bentiss>
mupuf: it's fire and forget, so it should be fine
<mupuf>
but you can;t start gitlab until it is done, right?
<bentiss>
which is way better than me babysitting the connections between the two cluster very 2h
<bentiss>
I can probably restore most of the data and then start gitlab
<bentiss>
some repos will return 404 but that shouldn't impact the rest
<mupuf>
Ack
Haaninjo has quit [Ping timeout: 480 seconds]
<mupuf>
bentiss: restarting Mesa CI will be fun
systwi has quit [Ping timeout: 480 seconds]
systwi has joined #freedesktop
<MrCooper>
bentiss: getting only 504s with git via https
<MrCooper>
tried xorg/proto/xorgproto and xorg/xserver so far
lack has joined #freedesktop
<bentiss>
MrCooper: yes, see above, In the end I ĺl do the full gitlab migration today I think
FireBurn has joined #freedesktop
tnt has joined #freedesktop
<bentiss>
turns out that trying to restore the main db was a good idea, I should have deleted the postgres disk on the new server where I did my tests first
Haaninjo has joined #freedesktop
kisak has joined #freedesktop
systwi has quit []
systwi has joined #freedesktop
<kisak>
morning, fwiw, I'm seeing 504 gateway time-outs over here when trying to get to gitlab.fd.o.
<daniels>
kisak: yeah, see all the messages above
daniels changed the topic of #freedesktop to: GitLab is currently down for upgrade; will be a while before it's back || https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
<kisak>
Oh right, parrot brain forgot the upgrade notice I saw yesterday. Sorry for the noise.
* alatiera
sends warm coffee over to admins
gbissett has joined #freedesktop
<bentiss>
oops... I actually tried to restore the db on the old cluster... which means this is completely failing... nothing to worry about FWIW
<alatiera>
I am sure we all wouldn't mind if you accidentally wiped all the open issues 😆
ximion has joined #freedesktop
gbissett has quit [Quit: Page closed]
jolley has joined #freedesktop
jolley has quit []
gbissett has joined #freedesktop
nedko has joined #freedesktop
lazka has joined #freedesktop
Ndfkjhw4 has joined #freedesktop
jsto has quit [Quit: jsto]
jsto has joined #freedesktop
<mupuf>
3 people so far reporting the 504 after 8h of outage. Sundays are good days for upgrades
<lazka>
my CI is failing, but I learned to read the channel topic last time :)
<bentiss>
damn... second time the db backup fails (with postgres just hanging)
<bentiss>
I'll try to restore both the db on the new and old cluster this time
jsto has quit [Quit: jsto]
jsto has joined #freedesktop
heapify has joined #freedesktop
heapify has quit [Remote host closed the connection]
JerryXiao has joined #freedesktop
gbissett has quit [Remote host closed the connection]
just_a_user has joined #freedesktop
<just_a_user>
hello, folks. gitlab.freedesktop.org is down?
<just_a_user>
Who knows of any mirrors?
<kisak>
See topic, scheduled upgrade in progress.
<just_a_user>
thanks a lot
gbissett has joined #freedesktop
<bentiss>
I think I found out why the db was not restoring properly: if someone accesses it while it's restoring, postgres just hangs... and this makes the restore fail
TrinitronX has joined #freedesktop
Inky1003 has joined #freedesktop
<bentiss>
(and yes, I had one rogue gitlab-exporter pod around)
<bentiss>
but that also means I can not restore gitaly while I restore the db, so I might not do the cold gitaly restoration and have to the the manual hot one :(
<Inky1003>
What happened to freedesktop's gitlab?
<Inky1003>
I needed to see wireplumber docs, but It seems It's down :(
<Inky1003>
Oh sorry, I haven't saw the description
<Inky1003>
nvm
Inky1003 has quit [Quit: Page closed]
keypresser86 has joined #freedesktop
<mupuf>
bentiss: oh no, 2 weeks of pain then
<bentiss>
mupuf: yeah... or I can try to do something otherwise, but right now it's that way :(
<bentiss>
anyway, so far the restoration is working, and I'm past the couple of previous failures, so that's a good sign
<bentiss>
mupuf: also TBH, gitaly-0 failed its last backup when I messed up the old db, so I don't have the latest shiniest data on those in a backup
just_a_user has quit [Quit: Page closed]
<mupuf>
bentiss: hopefully not much of value will be lost
<mupuf>
And it's git: just re-push
kisak has left #freedesktop [#freedesktop]
gbissett has quit [Quit: Page closed]
i-garrison has quit []
mirai has quit [Ping timeout: 480 seconds]
jarthur has joined #freedesktop
lack has quit [Read error: Connection reset by peer]
lack has joined #freedesktop
i-garrison has joined #freedesktop
agd5f has joined #freedesktop
agd5f_ has quit [Ping timeout: 480 seconds]
mirai has joined #freedesktop
AbleBacon has joined #freedesktop
An0num0us has joined #freedesktop
<bentiss>
the db restoration on the new cluster is still crashing. Rebooting one node that is showing issues. Meanwhile, the restoration on the old cluster is litterally taking ages
<meghand>
is there a web page explaining what the gitlab maintenance is doing?
killpid_ has joined #freedesktop
<bentiss>
meghand: no, sorry. long story short: things did not went as well as expected and it's still not over
<bentiss>
alright status update: both db backup (new/old clusters) are still going on. The new db just passed ~50% in 2h20 min, when the old cluster one is ~60% done in 6h of time
<bentiss>
I definitely be asleep when this finishes, so that means no gitlab on Monday morning for everyone (especially the .au guys)
sima has quit [Ping timeout: 480 seconds]
co1umbarius has quit [Ping timeout: 480 seconds]
co1umbarius has joined #freedesktop
Leopold has quit []
<DavidHeidelberg[m]>
bentiss: good night and sleep well :)
Leopold_ has joined #freedesktop
<karolherbst>
thanks for all the great work and dealing with all of this. Hopefully things will go smoothly tomorrow
<emersion>
♥
alyssa has joined #freedesktop
<alyssa>
++
rpigott has joined #freedesktop
<daniels>
bentiss: thankyou & sleep well!
callum_ has joined #freedesktop
An0num0us has quit [Ping timeout: 480 seconds]
callum_ has quit []
zondino has joined #freedesktop
<zondino>
What is meant by “while”? It’s already been a while. I cannot get working desktop without Libinput, where I can download v-1.24.0?