daniels changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
ngcortes has quit [Read error: Connection reset by peer]
Kayden has joined #freedesktop
co1umbarius has joined #freedesktop
columbarius has quit [Ping timeout: 480 seconds]
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
ximion has quit [Quit: Detached from the Matrix]
swick[m] has joined #freedesktop
gnfzdz[m] has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
nnm has quit []
nnm has joined #freedesktop
MrBonkers has joined #freedesktop
MrBonkers has quit []
MrBonkers has joined #freedesktop
i509vcb has quit [Quit: Connection closed for inactivity]
i509vcb has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
M0xMRTTOld[m] has quit [Ping timeout: 480 seconds]
MrBonkers has quit []
MrBonkers has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
seaweed[m] has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
pobrn has quit [Ping timeout: 480 seconds]
lkundrak has quit [Remote host closed the connection]
lkundrak has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
pitch[m] has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
nazarewk[m] has joined #freedesktop
MrBonkers has joined #freedesktop
ttancos[m] has joined #freedesktop
damian has joined #freedesktop
sima has joined #freedesktop
Inline has joined #freedesktop
marcel203s[m] has joined #freedesktop
havdan[m] has joined #freedesktop
nielsdg has joined #freedesktop
i509vcb has quit [Quit: Connection closed for inactivity]
Leopold__ has joined #freedesktop
Leopold_ has quit [Remote host closed the connection]
tjaalton has left #freedesktop [#freedesktop]
tinywrkb has joined #freedesktop
<mupuf>
bentiss: seems like we still have some broken kernel trees, coming from after the migration (if I understood the above correctly)
thaller has quit [Ping timeout: 480 seconds]
<bentiss>
FWIW I found out yesterday night what was wrong with oauth (and some other issues): debian now uses systemd-resolved, and ther e is a broken script that adds "DOMAINS" to the search of /etc/resolv.conf
<bentiss>
it works fine on the host, but the containers take that, and any request (foo.com for example) gets resolved to "foo.com.DOMAINS", which resolved to cloudfront
<bentiss>
and that messes up SSL for obvious reasons
<bentiss>
it should be fixed now, but I'll reboot all the nodes later today
<bentiss>
(restarting the gitaly pods to solve some push mirrorring issues)
Trevinho[m] has joined #freedesktop
gfxstrand has quit [Ping timeout: 480 seconds]
gfxstrand has joined #freedesktop
mvlad has joined #freedesktop
Juest has quit [Read error: Connection reset by peer]
Juest has joined #freedesktop
linyaa has quit [Read error: Network is unreachable]
linyaa has joined #freedesktop
<bentiss>
starting to reboot the servers, with server-15 being the first. It's the bridge between the 2 clusters, so chances are that there might be some service interruption
ManMower has quit [Remote host closed the connection]
ManMower has joined #freedesktop
alanc has quit [Ping timeout: 480 seconds]
dcbaker has joined #freedesktop
alanc has joined #freedesktop
bilboed7 has joined #freedesktop
<bentiss>
sigh, the fix from that stackexchange issue did not work. Working on it
cmeissl[m] has joined #freedesktop
bilboed has quit [Ping timeout: 480 seconds]
bilboed7 is now known as bilboed
jtatz[m] has joined #freedesktop
AbleBacon has quit [Read error: Connection reset by peer]
<bentiss>
hakzsam: you mean from gitlab to you or the other way around?
<hakzsam>
gitlab to me, yes
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
<bentiss>
I'm rebooting the server nodes, so maybe that's why, but I still see emails going out
<hakzsam>
okay, thanks
MrBonkers has joined #freedesktop
jenatali has joined #freedesktop
pac85[m] has joined #freedesktop
<bentiss>
FWIW, gitaly-1 was on a server that is showing some issues to restart, so there s part of the service down, should be back up in a couple of minutes
bendlas[m] has joined #freedesktop
blatant has joined #freedesktop
DodoGTA has quit [Quit: DodoGTA]
DodoGTA has joined #freedesktop
thaller has joined #freedesktop
lynxis has quit [Remote host closed the connection]
lynxis has joined #freedesktop
mchehab_ has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
<bentiss>
\o/ starting the reboot process of teh last server
Haaninjo has joined #freedesktop
shashanks_ has quit [Remote host closed the connection]
<svuorela>
is there a two liner on what happened since people are asked to re-push things ?
Inline has quit []
<bentiss>
svuorela: I changed the datacenter, made a mirror copy of the git data on Thursday thinking that the data will be synchronized, switched to the new datacenter, realized that the datra was not correctly synchronized
<svuorela>
oh.
<bentiss>
svuorela: and the original data got destroyed in the process, we do have a backup of Monday morning, so worse case we can dig into it,but for 99% of the case, someone just needs to push again their changes..
<bentiss>
and for the past 24h I've spent my time fixing all sort of mishaps. But I think it's almost good now :)
<pq>
\o/
<svuorela>
it was just in some channel people were trying to find out if anyone had certain changesets, I think they were found.
<svuorela>
but some swearing was done in the process. But probably less swearing than what you did when you discovered the mishap.
<bentiss>
svuorela: I was told a good answer to that: "please make them join the admin team so they can spend their own week and weekend to restore all of the git data for anyone" :)
<bentiss>
it's a pain, I messed up, but I still believe that process was way better than me spending a month hand-holding the migration of the live data
<bentiss>
(and TBH, if teh bulk is now done, I still have a bunch to do: registry, bots, and some other services like indico)
<svuorela>
I've no clue about the amount of time and data but with my amateur skills, I would have synced stuff. turned to read-only mode, synced again (hoping this would be fast), swapped over, reenabled things.
shashanks has joined #freedesktop
<svuorela>
but whatever was lost was actions happened between thursday and yesterday morning ?
<bentiss>
svuorela: turning to readonly is.... hard. And then synchronizing the data is also hard
<bentiss>
svuorela: yes, that's the time frame
<bentiss>
and not "action" git push/merges
<bentiss>
all of teh db is fine
<bentiss>
so MRs, comments, issues, jobs logs
<svuorela>
also git merges from webui or only "manual" git actions ?
<bentiss>
svuorela: any git data that was pushed, so if the data was there on Wednesday, and got merged on Sunday, the data is still there, and the branch might just need to be toppedup
<bentiss>
anyway, lunch time for me
pobrn has joined #freedesktop
shashanks has quit [Remote host closed the connection]
<mupuf>
bentiss: sorry, I was out until now. Is the 3rd party login fixed?
<mupuf>
You mentioned you figured out what the problem was, then said the fix didn't work. Not sure what's the state of it now :)
<mupuf>
And finally, I got quite a few "WARNING: Uploading artifacts as "archive" to coordinator" during the morning... but that may be while you were restarting the servers
Nero has joined #freedesktop
<daniels>
mupuf: yep, third-party login is working now
<mupuf>
great, thanks. So no need to update the banner
<mupuf>
I made it dismissable now
msizanoen[m] has joined #freedesktop
<bentiss>
mupuf: yeah, I found another fix for the script, deployed and rebooted all servers
<bentiss>
we are now using the new servers and have an IPv6, this is so much the future :)
<mupuf>
oh boy, oh boy indeed!
Hazematman has joined #freedesktop
<nirbheek_>
Is it just me or is gitlab loading really slowly? It's taking minutes to load an issue, and it's incomplete at the end of it
<bentiss>
nirbheek_: probably the cache needs to warm up
AnuthaDev has joined #freedesktop
<nirbheek_>
Okay, thanks :)
mitTengiz[m] has joined #freedesktop
hololeap has quit [Quit: Bye]
hololeap has joined #freedesktop
<cwabbott>
DavidHeidelberg: seems there's a fun bug with ci_run_n_monitor.py where it always cancels the sanity job, so if you run it before that finishes then it gets stuck
<sima>
bentiss, yay the future, and thanks for all the hard work you're putting in!
pobrn has quit [Remote host closed the connection]
<daniels>
^
lsd|2 has joined #freedesktop
AnuthaDev has quit []
lsd|2 has quit []
lsd|2 has joined #freedesktop
AnuthaDev has joined #freedesktop
<DavidHeidelberg>
^²
<DavidHeidelberg>
cwabbott: before it was probably always fast enough to not hit it :D I think Guilherme send MR w/ fix
shashanks has joined #freedesktop
vkareh has joined #freedesktop
AnuthaDev has quit []
lsd|2 has quit []
lsd|2 has joined #freedesktop
agd5f_ has joined #freedesktop
tomeu has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
agd5f has quit [Ping timeout: 480 seconds]
MrBonkers has joined #freedesktop
MrBonkers has quit []
MrBonkers has joined #freedesktop
AnuthaDev has joined #freedesktop
blatant has quit [Quit: WeeChat 4.1.0]
dabrain34[m] has joined #freedesktop
duncaen has quit [Remote host closed the connection]
duncaen has joined #freedesktop
<linyaa>
question about imagination vulkan testing in the Mesa gitlab CI. are the machines hosted externally by IMG, similar to how Google/ChromeOS hosts many qualcomm devices in the CI?
<daniels>
linyaa: fd.o doesn't host any hw-testing devices itself, just generic x86-64/aarch64 runners for builds
<daniels>
I haven't seen IMG testing yet be enabled, but it would either be in a lab that they've set up and hosting themselves, or Collabora's which has a load of devices