<emersion>
ishitatsuyuki: maybe consider getting involved with the sysadmin stuff if you're annoyed
<emersion>
in other words: nobody is entitled to GitLab uptime guarantees, this is best-effort and run by volunteers
aleksander has joined #freedesktop
<shadeslayer>
MrCooper: ack, thanks
<bentiss>
daniels: I am looking currently at the 2 crashes we had overnight. the 502 we get are correlated with the OSD getting killed because we are at 100% memory usage (by comparing the 502 on kibana and the graphs from influxdb)
<bentiss>
daniels: so maybe it's not entirely network related, but also ceph eating too much memory (15GB per osd when you have 13 disks on a machine eats up most of the memory)
ximion has joined #freedesktop
MrCooper has quit [Quit: Leaving]
MrCooper has joined #freedesktop
ximion has quit []
<bentiss>
daniels: interesting... I am trying to map our cluster to https://github.com/rook/rook/issues/5811 -> it seems our osd_memory_target is 216 GB per OSD on the HDDs, I guess we ought to set the limits and requests in the cluster config
<bentiss>
FTR: in the ceph toolbox `ceph tell osd.25 config show | jq '.osd_memory_target'` returns 216297281945
* bentiss
is applying saner limits to the OSD, in the hope that they won't be OOM-killed like they are today
<bentiss>
daniels: ceph tell osd.25 config show | jq '.osd_memory_target' => 4294967296 much more reasonable
<shadeslayer>
I see, I'd like to make a change to Mesa CI going forward to use the commit sha instead of the pipeline id, since it makes more sense to artifact the build there
<bentiss>
shadeslayer: you can have more that one pieline per sha, how are you going to solve that?
<shadeslayer>
the sha + artifact name is unique?
<bentiss>
yes, but if you retry the job, you'll overwrite the data
<shadeslayer>
Ah, I see
<bentiss>
actually, using the pipeline ID is having the same issue...
<bentiss>
shadeslayer: also, the reason why I relied on pipeline id is because that is stored in the JWT token, while the sha might not (there is a ref, but it's not guaranteed to be a sha)
<bentiss>
don't get me wrong, I am opened to suggestions, but I'd rather have some sort of namespace for enforcing the artifacts path, or a very good reason to not enforce it (besides at the project path level)
<shadeslayer>
bentiss: well, it's using the JOB_ID, so it should be fine
<shadeslayer>
The reason I wanted to use the sha was to be able to have a immediate reference to the built sha in the CI YAML
<shadeslayer>
instead of a obscure job id
<bentiss>
shadeslayer: it's using the pipeline id, not the job, so you can gather all artifacts of one pipeline in one place
<bentiss>
but honestly, I am not sure there are many users besides mesa, so we should be able to add a new rule easily
<shadeslayer>
ah yeah, I was looking at JOB_ARTIFACTS_BASE
<shadeslayer>
which uses JOB_ID
<bentiss>
shadeslayer: but OTOH, why are you using minio-packet for that? you could rely on regular artifacts and it won't induce a cost for us
<shadeslayer>
Ah I see, I'll just drop that then
raghavgururajan has quit [Quit: Gateway shutdown]
raghavgururajan has joined #freedesktop
linguist has joined #freedesktop
linguist has left #freedesktop [#freedesktop]
autistic has joined #freedesktop
brunodOut has joined #freedesktop
autistic has quit [Ping timeout: 480 seconds]
Haaninjo has quit [Quit: Ex-Chat]
Haaninjo has joined #freedesktop
Haaninjo has quit [Remote host closed the connection]
Haaninjo has joined #freedesktop
Haaninjo has quit [Remote host closed the connection]
Haaninjo has joined #freedesktop
bookworm has joined #freedesktop
alatiera has joined #freedesktop
ngcortes has joined #freedesktop
alatiera is now known as Guest6808
d_ed has quit [Ping timeout: 480 seconds]
raghavgururajan has quit [Remote host closed the connection]
raghavgururajan has joined #freedesktop
raghavgururajan has quit [Remote host closed the connection]
ngcortes has quit [Read error: Connection reset by peer]
raghavgururajan has joined #freedesktop
ngcortes has joined #freedesktop
raghavgururajan has quit [Quit: Gateway shutdown]
raghavgururajan has joined #freedesktop
ngcortes has quit [Ping timeout: 480 seconds]
ngcortes has joined #freedesktop
shbrngdo has quit [Remote host closed the connection]
shbrngdo has joined #freedesktop
danvet has quit [Ping timeout: 480 seconds]
shbrngdo has quit [Read error: Connection reset by peer]
shbrngdo has joined #freedesktop
pendingchaos_ has joined #freedesktop
pendingchaos has quit [Read error: No route to host]
d_ed has joined #freedesktop
pendingchaos_ is now known as pendingchaos
ngcortes has quit [Ping timeout: 480 seconds]
ndufresne has quit [Quit: Ping timeout (120 seconds)]
fahien2 has quit []
ocrete has quit [Quit: Ping timeout (120 seconds)]
tomeu has quit [Quit: Ping timeout (120 seconds)]
shadeslayer has quit [Quit: Ping timeout (120 seconds)]
ocrete has joined #freedesktop
fahien2 has joined #freedesktop
ndufresne has joined #freedesktop
ndufresne is now known as Guest6829
tomeu has joined #freedesktop
shadeslayer has joined #freedesktop
ngcortes has joined #freedesktop
d_ed has quit [Ping timeout: 480 seconds]
Seirdy has quit [Ping timeout: 480 seconds]
Seirdy has joined #freedesktop
ngcortes has quit [Remote host closed the connection]