10:09 <bentiss> daniels, anholt: FYI, I just put in prod a registry-cache that will be auto-pruned every Sunday

10:10 <bentiss> so in theory, we shouldn't have to kick it anymore

10:10 <bentiss> roughly https://gitlab.freedesktop.org/freedesktop/helm-gitlab-omnibus/-/commit/e945558c9cf91718737f180baea7f2fbf9340513 (and the next one)

10:11 <bentiss> daniels, anholt: the thing is now the registry cache is in its own namespace: "registry-cache", so do not look for it in the gitlab namespace

10:13 <daniels> nice, thankyou!

10:13 <bentiss> hopefully this will work, we'll see Sunday :)

10:15 <bentiss> daniels: while you are there: https://gitlab.freedesktop.org/gfx-ci/mesa-performance-tracking/-/jobs/14529611 all of the "upload-data" jobs are failing

10:18 <bentiss> not sure what is going on, ci-stats-influxdb2.freedesktop.org is working here (though is quite slow)

10:52 <thaller> hi. NetworkManager's gitlab-ci pipelines run always into "Job's log exceeded limit of 4194304 bytes." ... that limit seems quite small, would it be possible to significantly bump it?

10:53 <thaller> (usually, if there is a problem, then the test will fail early, when we still have logs. But not always, and having the logs truncated, is a bit annoying)

10:54 <thaller> if storage is the problem, then we can delete older pipelines eagerly. We almost never case about pipelines from a few days ago (or older)

10:56 <thaller> (biab)

10:56 <pq> Would it not be a browser DoS if someone went to look at a successful job?

11:03 <__tim> that's just for the output isn't it? Maybe you could dump failure details into log files and upload them as artefacts instead?

11:32 <thaller> pq, the javascript view of the log, only shows the tail anyway. You'd have to click to look at the raw files... where ok, I guess you are right.

11:32 <thaller> I think my browser could handle 20mb easily...

11:33 <thaller> __tim, yes, alternative solutions are possible... also, we have so much output, because we run so many things in one job. we could just make more, smaller jobs (but we already have plenty of jobs, not sure more jobs is the solution).

11:59 <pq> oh, right, gitlab only shows the last 0.5 MB of the log in the web page

14:47 <agd5f> is something up with gabe.freedesktop.org? I keep getting errors returned when I reply to mailing lists hosted there

14:47 <agd5f> looks like the disk is full on gabe

14:48 <agd5f> daniels, ajax ^^

14:48 <bentiss> emersion: ^^

14:49 <emersion> ah, this again

14:49 <bentiss> yeah, can't remember what you did last time :(

14:54 <ajax> i'm not sure what i _can_ do

14:55 <ajax> but yeah, /home is full

14:55 <ajax> / isn't so hopefully services kinda work but

14:57 <ajax> bentiss: can you dump a 'du -h /home | sort -nk1' somewhere? that machine really shouldn't be doing much of anything, if we're out of 150G of /home someone needs some gentle correction

14:57 <ajax> sudo in the obvious place

14:59 <bentiss> IIRC the ML logs/archives are on /home

15:03 <emersion> there's ncdu as well

15:03 <emersion> yeah archives are in /home

15:10 <jekstrand> daniels, bentiss: Looks like someone needs to kick the mailman machine. It thinks it's out of disk.

15:10 <jekstrand> Which might be what you're already talking about...

15:10 <emersion> yup

15:10 <bentiss> jekstrand: exactly :)

15:10 <emersion> looking into it rn

15:11 <jekstrand> Cool, thanks!

15:53 <demarchi> intel-gfx@lists.freedesktop.org

15:53 <demarchi> Your message couldn't be delivered. Try to send it again later. If the problem continues, please contact your email admin.

15:53 <demarchi> intel-gfx@lists.freedesktop.org

15:53 <demarchi> Remote Server returned '554 5.3.0 < #5.3.0 x-unix; Traceback (most recent call last): File "/var/lib/mailman/scripts/post", line 69, in <module> main() File "/var/lib/mailman/scripts/post", line 64, in main tolist=1, _plaintext=1) File "/var/lib/mailman/Mailman/Queue/Switchboard.py", line 140, in enqueue fp.flush() IOError: [Errno 28] No space left on device>'

15:53 <demarchi> something wrong in the server...

15:55 <bentiss> demarchi: yes we are aware, emersion is looking into it

16:00 <emersion> hm so there's /home/cvs-old/cvs

16:01 <emersion> daniels said we should tar it and archive it on annarchy

16:01 <emersion> but i don't have access to annachy

16:01 <emersion> /home/jg/public_html/xorg.tar.gz is taking 250MiB

16:01 <bentiss> me neither

16:01 <emersion> /home/cworth/mail-logs is 500MiB

16:02 <emersion> ah hm that one i already got rid of last time

16:02 <emersion> (still scanning, so reading the logs from last time)

16:11 <demarchi> bentiss: emersion thanks

16:12 <emersion> hmmm so

16:12 <emersion> 13.7 GiB [##########] /libreoffice-bugs

16:12 <emersion> 7.3 GiB [##### ] /libreoffice-commits

16:12 <emersion> 7.3 GiB [##### ] /gstreamer-commits

16:13 <emersion> let's rm bounce-events as well, 1GiB

16:14 <emersion> /home/srv/bod.x.org/archives.git is 500MiB

16:16 <emersion> daniels, what would it take to get more disk space?

16:16 <emersion> i though a few GiBs was plenty, but apparently not

16:18 <emersion> well if we could get ird of the *-commits mailing lists that'd already be nice

16:19 <emersion> in 2021-07-20 we have 3GiB of free space

16:19 <emersion> had*

16:21 <emersion> so in ~1 month we'll run out of space again

16:23 <__tim> I'm surprised the commits lists are archived actually

16:25 <__tim> I can find the first commit from the gstreamer list archive in the git logs, so think the history is all there for us

16:26 <__tim> guess the only advantage is that stuff in the list archive might be found via google?

16:30 <__tim> nope, not even that

16:35 <__tim> I have turned off archiving of messages for gstreamer-commits now

16:46 <daniels> I can actually grow the underlying block device

16:46 <daniels> doing that now

16:47 <daniels> ETA 14min apparently

17:13 <emersion> ah nice

17:18 <daniels> just rebooting it so then we can resize

17:25 <daniels> done

17:36 <emersion> 21G

17:36 <emersion> sweet

17:38 <emersion> can i disable archives for libreoffice-bugs?

17:38 <emersion> it's all stored in bugzilla as well

18:00 <daniels> yeah, I think binning it is completely fine - when they wanted to move their lists last year they didn't seem to care about bugs