ChanServ changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
scrumplex_ has joined #freedesktop
scrumplex has quit [Ping timeout: 480 seconds]
ihsnhsshsb[m] has joined #freedesktop
ihsnhsshsb[m] has left #freedesktop [#freedesktop]
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
JanC is now known as Guest11696
JanC has joined #freedesktop
Guest11696 has quit [Ping timeout: 480 seconds]
eluks has quit [Remote host closed the connection]
ximion has quit [Remote host closed the connection]
kode54 has joined #freedesktop
K`den is now known as Kayden
<mupuf>
bentiss: any news regarding fastly?
<mupuf>
oh, and the migration page is lacking quite a few ticks now
<bentiss>
mupuf: unfortunately no
<bentiss>
and on the good news, the high load onthe db stopped at 1:30 AM CET+1 (over the night) and the system is much more stable now
<bentiss>
It would seem splitting the db triggered some re-indexing or something like that
<bentiss>
well, we should probably wait for the US to wake up too, but the fact that the load dropped very suddenly makes me thing of a process finishing its job
<mupuf>
that is indeed amazing news!
<bentiss>
yeah, no need to buy more expensive HW :)
<mupuf>
yet* 😃
<bentiss>
heh, yes
<bentiss>
well, I think we should also enforce the 1 year expiration policy on pipelines in all projects
<bentiss>
this should reduce the size of the db by a fair chunk
<mupuf>
absolutely!
<bentiss>
(and it's working blazing fast now)
<mupuf>
there isn't much value in those
<bentiss>
re: oh, and the migration page is lacking quite a few ticks now -> I'm mostly using this as a TODO list, so as I think at new things, I'm just adding them here
<gachikuku>
Is there a way that if a url has youtube in it then make xdg-open use mpv or else browser?
sghuge has quit [Remote host closed the connection]
sghuge has joined #freedesktop
jsa1 has joined #freedesktop
swatish2 has quit [Ping timeout: 480 seconds]
AbleBacon has quit [Read error: Connection reset by peer]
tzimmermann has joined #freedesktop
<slomo>
bentiss: gitlab is super fast now, good work :)
<bilboed>
I'm here to complain that gitlab is too fast ! I used to have an excuse for not being so productive and it's gone !
<kode54>
thanks for the super job migrating this glorious thing
<dj-death>
yeah, great it's back so fast, great job everybody involved
<bilboed>
yah, more seriously : Awesome job !
<ofourdan>
alanc: fwiw, I am not receiving any email from fd.o either
<ofourdan>
(and that's on gmail, not my work email)
<ofourdan>
could be normal though, only update for the xserver I see is metux pushing new commits to their MR xorg/xserver!1865 and I usually don't get notified every time people pish new commtis to an existing MR (thanksfully!).
<ofourdan>
s/pish/push/
kasper93 has quit [Ping timeout: 480 seconds]
<bentiss>
alanc: and the IP didn't change, it's stil gabe.fd.o the smtp server
<bentiss>
(FWIW, I received emails from gitlab with failed/fixed pipelines as the db server was overloaded)
sima has joined #freedesktop
<mupuf>
same here
<soreau>
was surprised to see gitlab working and seems faster than before
<soreau>
great work
<pq>
bentiss, thank you for the on-going hard work and the awesome maintenance page. Gitlab being out allowed me to be productive myself, so I think I'll continue so for the rest of the week. ;-)
mripard has joined #freedesktop
<bentiss>
thanks everybody ;)
<mripard>
congrats and thanks everyone
swatish2 has quit [Ping timeout: 480 seconds]
<soreau>
magnitudes faster 🚀 🎉 👍
<mripard>
yeah, it's super fast
<bilboed>
pq: I knew I wasn't the only one thinking that :D
<fomys>
Thanks a lot for the migration! Did you change anything during the migration, it feels very fast, Congratulation!
kasper93 has joined #freedesktop
<slomo>
bentiss: ooc, gnome gitlab has anubis put in front of it for making the problem of AI scrapers smaller. no idea how well that works but as it's there since quite a while now it probably has some impact at least. do you plan to do something similar for fdo?
swatish2 has joined #freedesktop
<MrCooper>
alanc: FWIW, I saw him pushing to existing MRs, not creating new MRs
<MrCooper>
personally I'd be less worried about Git (can always just push again if something gets lost) than about new MRs / issues / comments / ...
<bentiss>
kxkamil: yesterday I would have said no, today it's much better. But I'd like to see a couple of days where the US kicks in. So I would say you can push changes, just keep in mind what you did, and do not rely on issues, MRs, etc...
<bentiss>
slomo: we should have Fastly as a CDN soon(tm), once this is set up, we won't need anubis because they have bots/spam protection
<bentiss>
fomys: almost no changes: we just went from a single db node hosted on kubernetes with a ceph disk to 3 dedicated HA postgresql cluster with actual NVME disk. Very little impact as you can see :)
<jadahl>
In the welcome to the new data center note, it says we should consider it roughly "read-only". does that mean we should e.g. create releases and expect them to be there once the migration is completed?
<jadahl>
should *not* create releases
<bentiss>
more seriously, I'm glad we managed to pin point the pain point :)
<mripard>
bentiss: so kubernetes and ceph is a conspiracy by Big HW to sell you more HW?
<bentiss>
jadahl: if you can refrain from creating releases during a couple of days that would be better
<bentiss>
mripard: no, it's just that your admin sys doesn't know what to do and thought the postgresql db was just a simple part, not an actual important part
<jadahl>
bentiss: noted, thanks!
<bentiss>
again, yesterday we had an average load on the db servers of 30-40 on 12 cores machines
<bentiss>
today it's fine (between 1 to 2) but I'd like to see what happens when the US start hammering our servers
<fomys>
bentiss: I see, thank you for this big improvement! I hope you will not encounter issue with the remaining tasks
<DragoonAethis>
bentiss: let us enjoy the speed while it lasts :^)
swatish21 has joined #freedesktop
swatish2 has quit [Ping timeout: 480 seconds]
swatish21 is now known as swatish2
<bilboed>
Maybe some tariffs are in order for US traffic ?
<Venemo>
hey guys, very nice work on the migration
* bilboed
runs away because of bad joke
<Venemo>
haha
enunes has joined #freedesktop
guludo has joined #freedesktop
swatish2 has quit [Ping timeout: 480 seconds]
alarumbe has joined #freedesktop
GNUmoon2 has quit [Remote host closed the connection]
GNUmoon2 has joined #freedesktop
wontfix[m] has joined #freedesktop
haaninjo has joined #freedesktop
<bentiss>
daniels: mind if I had an aggressive pipeline expiration policy on gfx-ci-bot/dummy-gitlab-jobs? this project has 73939 pipelines alone ;)
<bentiss>
*if I add
<daniels>
bentiss: omg ... yes, please
<daniels>
they can expire after 1d
<bentiss>
cool
<bentiss>
FWIW, the expiration policy is blazing fast compared to previously, so we can add more projects
<daniels>
\o/ \o/
<bilboed>
how's the load holding with US waking up ?
<bentiss>
still not processed the remaining 98227 pipelines from mesa, but over the past couple of weeks at equinix, we had like 5000 pipelines cleaned, while here, since yesterday, we are at 20000 down
<bentiss>
daniels: gfx-ci/mesa-performance-tracking is at 132432 pipelines... how much retention we want?
<mupuf>
bentiss: these projects do not require more than a week
<wontfix[m]>
Is there a better place to ask about the internals of a specific small freedesktop project?
<mupuf>
tanty's proejcts can be dropped, quite likely
<zmike>
are those pipelines that have actually been run?
<zmike>
or just creatred
<bilboed>
nirbheek: cerbero is in that list that bentiss pasted above. What amount of retention (in days) for past pipelines would we reasonably need ?
<bentiss>
zmike: don't know if the pipeline ran, but it still takes some sapce in the db :)
<zmike>
I would say delete any pipelines that have not been run after 2-3 days at most
<bentiss>
bilboed, nirbheek: this can be a year, I'm fine with that
<bentiss>
hakzsam: I'm trying to purge the CI database with old pipelines to make it smaller and faster. Do you need to keep all of those pipelines or can we add a retention policy of 1 year (or less)
<hakzsam>
bentiss: I don't need them, a week should be enough
<bentiss>
hakzsam: thanks!
<bentiss>
zmike: I can not conditionally prune, so I put 1 year in the field, this should remove 11805 out of the 12892 :)
<bentiss>
daniels: I assume gfx-ci/igt-ci-tags can also be like a couple of weeks?
<daniels>
bentiss: yeah
<daniels>
though tbh that service should just be moved to some intel git server somewhere
<bentiss>
OK, done :)
<bentiss>
thanks everyone who responded
<daniels>
thanks!
<DragoonAethis>
daniels, bentiss: Yeah, igt-ci-tags et al should be moved elsewhere to avoid burning gitlab.fd.o resources
<DragoonAethis>
Both on pipelines and tags tbh
<DragoonAethis>
We have some code pushing resources there, but don't know who else might be using the results there
<bentiss>
heh... 501597 pipelines scheduled for deletion, out of a grand total of 1281274 -> this alone should shrink the CI db by a fair bit
<DragoonAethis>
bentiss: do you have a way to set a global retention policy?
<DragoonAethis>
Something like 3 months unless you ask for more/less per project
<bentiss>
DragoonAethis: in the long run I'll run a script to set to one year (the retention of the artifacts now)
<bentiss>
should be doable with rails, but rn, easier for me to edit the URL, and change the 10 projects I'm focusing on
<DragoonAethis>
I can appreciate healthy amounts of duct tape too ;)
<bentiss>
damn... PG errors: out of memory :(
<bentiss>
don't knwo if it comes from sidekiq or the db itself...
<__tim>
minor issue, but not sure if the approve-users bot/webhook is working
* bentiss
looks
<bentiss>
oops, invalid token
<bentiss>
restarting them (I had the same issue with marge, except marge crashes on boot if the token is invalid)
<bentiss>
__tim: thanks :)
<__tim>
works now, thanks
<bentiss>
(for retrying the labels)
<__tim>
ah
<bentiss>
daniels: any luck with the runners so far?
<bentiss>
(to know if I should jump start on these)
<bentiss>
TBH I haven't realized how fast it is to push code now, I was just focusing on the web UI :)
<mripard>
I did realize how fast dim ub is now :)
<mripard>
it's really awesome
<daniels>
bentiss: I've not had the chance yet, sorry :( have been stuck with other stuff all week so far
<bentiss>
daniels: no worried
<bentiss>
worries
<bentiss>
I'm a little bit concerned about these "ActiveRecord::StatementInvalid: PG::OutOfMemory: ERROR: out of memory" dead jobs
<bentiss>
sigh... __vm_enough_memory: pid: 1054316, comm: postgres, no enough memory for the allocation in the journal of the leader PG
ximion has joined #freedesktop
<bentiss>
trying to reload the db...
<bentiss>
I've changed a sysfs parameter
<bentiss>
mayeb we'll need beafier machines :(
<bentiss>
I've reduced the number of sidekiq pods, this seemed to have an effect
<Kayden>
honestly I think 1m retetion instead of 1y for mesa would probably be fine
<bentiss>
currently I see a lot of background job failing with 404 about ""MergeRequest", 107226, "Ci::CompareTestReportsService", nil, nil" -> and this is comparing test reports with old MRs
<bentiss>
so it makes sense to keep a little bit the traces
<bentiss>
for personal repos, I guess we don't care
<Kayden>
huh. wonder what's doing that
<bentiss>
I think it's the test report summary in the MR
<Kayden>
ah
<bentiss>
it compares the previous pipeline report to tell you if things improved or not
<bentiss>
anway, as long as we keep the db to a constant size, that would be nice
<Kayden>
yep.
jsa1 has quit [Ping timeout: 480 seconds]
tzimmermann has quit [Quit: Leaving]
swatish2 has joined #freedesktop
swatish2 has quit [Ping timeout: 480 seconds]
AbleBacon has joined #freedesktop
<robclark>
daniels: might want to keep gfx-ci-bot pipelines around for a bit longer than a day... I go back and look at them once in a while.. but a week or two would be ok?
<daniels>
robclark: this is a project which was just a temporary workaround for an old gitlab-runner bug; the uprev pipelines will stay for as long as any other mesa/virglrenderer pipeline does
nirbheek_ has joined #freedesktop
<nirbheek_>
bentiss: you can get rid of all logs except the last 3 months tbh