ChanServ changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
random_james has quit [Remote host closed the connection]
random_james has joined #freedesktop
aswar002 has quit [Remote host closed the connection]
aswar002 has joined #freedesktop
ybogdano has quit [Remote host closed the connection]
ybogdano has joined #freedesktop
pzanoni has quit [Remote host closed the connection]
pzanoni has joined #freedesktop
ybogdano has quit [Ping timeout: 480 seconds]
ngcortes has quit [Remote host closed the connection]
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
mvlad has joined #freedesktop
vbenes has joined #freedesktop
Haaninjo has joined #freedesktop
MajorBiscuit has joined #freedesktop
eroux_ has joined #freedesktop
eroux has quit [Ping timeout: 480 seconds]
thaller has joined #freedesktop
thaller has quit [Read error: No route to host]
vbenes has quit []
vbenes has joined #freedesktop
fahien has joined #freedesktop
MrCooper has quit [Quit: Leaving]
MrCooper has joined #freedesktop
thaller has joined #freedesktop
ximion has joined #freedesktop
ximion has quit []
ximion has joined #freedesktop
fahien has quit [Ping timeout: 480 seconds]
fahien has joined #freedesktop
thaller is now known as Guest5614
thaller has joined #freedesktop
MajorBiscuit has quit [Quit: WeeChat 3.5]
MajorBiscuit has joined #freedesktop
Guest5614 has quit [Ping timeout: 480 seconds]
arinov has quit [Ping timeout: 480 seconds]
progandy has joined #freedesktop
ximion has quit []
GNUmoon2 has quit []
GNUmoon has joined #freedesktop
GNUmoon has quit [Remote host closed the connection]
GNUmoon has joined #freedesktop
arinov has joined #freedesktop
GNUmoon has quit [Remote host closed the connection]
GNUmoon has joined #freedesktop
Haaninjo has quit [Quit: Ex-Chat]
rgallaispou1 has quit [Read error: Connection reset by peer]
ybogdano has joined #freedesktop
___nick___ has joined #freedesktop
thaller is now known as Guest5629
thaller has joined #freedesktop
___nick___ has quit []
Guest5629 has quit [Ping timeout: 480 seconds]
___nick___ has joined #freedesktop
___nick___ has quit []
___nick___ has joined #freedesktop
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
arinov has quit [Ping timeout: 480 seconds]
thaller is now known as Guest5631
thaller has joined #freedesktop
Guest5631 has quit [Read error: Connection reset by peer]
arinov has joined #freedesktop
<karolherbst>
daniels: the nouveau wiki returns a 500 :(
<karolherbst>
something up with gitlab pages?
<tomeu>
also with artifacts uploading:
<tomeu>
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error id=25711390 responseStatus=500 Internal Server Error status=500 token=-_cdSo74
<daniels>
I can’t fix it until rather later when I’m back near my laptop, unless bentiss is around
ybogdano has quit [Ping timeout: 480 seconds]
<craftyguy>
oh I didn't see the reason above since I just rejoined. but if it's a known problem, then no need to repeat on my behalf :)
ybogdano has joined #freedesktop
<daniels>
given that I’m guessing the S3 gateway has taken a dive and is stubbornly staying in bed
<bentiss>
daniels: not sure I'll have time to work on it, but it seems we have one OSD full
<daniels>
bentiss: ahh, hmm - I’ll have to look up how to rebalance (?) when I get home then
<daniels>
or do you have a quick suggestion?
<bentiss>
daniels: problem is there is a high chance the whole pool is not accepting any command, and so it's going to be interesting to work on... (i.e. no ideas on how to quickly solve it)
<bentiss>
FWIW, I suspect a couple of reasons for having one disk full: backups (of course), and I have banned a few people building android on our runners, so that might grow the artifacts pool by an order quckly enough...
* bentiss
got to find food, bbl
<daniels>
:S
<daniels>
bon appetit!
arinov has joined #freedesktop
fahien has quit [Quit: fahien]
fahien has joined #freedesktop
fahien has quit []
vbenes has quit [Ping timeout: 480 seconds]
arinov has quit [Ping timeout: 480 seconds]
ngcortes has joined #freedesktop
<bentiss>
daniels: FYI, I run "ceph osd reweight-by-utilization" which is supposed to do some rebalancing
<bentiss>
daniels: and I just tried to force to reweight only the faulty OSD by running "ceph osd reweight-by-utilization 120 0.2 1". We should wait a few to see if that does anything
Haaninjo has joined #freedesktop
arinov has joined #freedesktop
<bentiss>
actually, "ceph osd reweight osd.1 1.0" allows for a targeted action if required
aswar002 has quit [Ping timeout: 480 seconds]
<bentiss>
I guess the best thing to do now is to wait for ceph to do its magic
<bentiss>
daniels: I think if it doesn't solve by itself we will have 2 options to try: mark osd.18 as down, it might be sufficient to restart the cluster, but we better flush the data from the disk before marking it as in (or it will be detected as full and will block)
<bentiss>
second option is to bring in a new machine in the cluster with HDD, and wait for it to settle before removing it
<bentiss>
FWIW, at the given pace, the misplaced data should be moved in ~6h, could be more, could be less, and maybe we'll need to do more recovery steps
<bentiss>
making progress: from "1 full osd(s)" to "1 backfillfull osd(s)"
<bentiss>
Seems to be back up (even one disk is still not in a good shape)
pabs has quit [Read error: Connection reset by peer]
ximion has joined #freedesktop
danvet has quit [Ping timeout: 480 seconds]
___nick___ has quit [Ping timeout: 480 seconds]
thaller has quit [Read error: Connection reset by peer]
thaller has joined #freedesktop
systwi has quit [Ping timeout: 480 seconds]
thaller is now known as Guest5651
thaller has joined #freedesktop
Guest5651 has quit [Read error: Connection reset by peer]
thaller has quit [Read error: Connection reset by peer]
arinov has quit []
progandy has quit [Ping timeout: 480 seconds]
bretttt has joined #freedesktop
Haaninjo has quit [Quit: Ex-Chat]
AbleBacon has quit [Read error: Connection reset by peer]