ChanServ changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
AbleBacon has joined #freedesktop
immibis has quit [Read error: Connection reset by peer]
numerator has quit []
ximion has quit []
airlied has joined #freedesktop
<airlied>
is there some issue with gitlab ssh?
<airlied>
seems to be having auth errors with my ssh key a lot, then it works sometimes
<airlied>
bentiss, daniels ^?
Rainer_Bielefeld has joined #freedesktop
agd5f has quit [Ping timeout: 480 seconds]
Rainer_Bielefeld has quit [Remote host closed the connection]
Rainer_Bielefeld has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
i-garrison has quit []
i-garrison has joined #freedesktop
<bentiss>
airlied: yesterday I had to reboot one of the nodes, maybe I'll need to reboot all of them. Was it on a particular repo?
<bentiss>
airlied: seems like one of the gitlab-shell pod is having issues, taking care of it
alatiera0 has joined #freedesktop
<airlied>
bentiss: it was nouveau/mesa repo, but I think I saw it on mesa/mesa as well
alatiera has quit [Ping timeout: 480 seconds]
<bentiss>
airlied: I'll just reboot the 2 nodes where shell was running
<bentiss>
I see errors from gitlab-shell, same ones that I saw on gitaly yesterday that were solved by a reboot
<bentiss>
I've rebooted large-5, now I need to reboot large-7, but I need to wait for ceph to be healthy again and the backup to finish first
danvet has joined #freedesktop
Rainer_Bielefeld has quit [Remote host closed the connection]
Rainer_Bielefeld has joined #freedesktop
Rainer_Bielefeld has quit [Remote host closed the connection]
<emersion>
it fails on one repo (emersion/libdisplay-info), but works on another one (wlroots/wlroots)
<emersion>
with just a `git pull`
<emersion>
libdisplay-info is on gitaly-3, wlroots on gitaly-2
Rainer_Bielefeld has joined #freedesktop
<emersion>
ah working again now
Rainer_Bielefeld has quit [Remote host closed the connection]
<__tim>
gstreamer marge also going crazy because it cant' git clone
<bentiss>
sorry guys, I'll have a look at it early in the afternoon (public holiday here and I'm not in front of the computer all day)
___nick___ has joined #freedesktop
immibis has joined #freedesktop
Rainer_Bielefeld has joined #freedesktop
AbleBacon has quit [Read error: Connection reset by peer]
kchibisov has quit [Read error: Connection reset by peer]
kchibisov has joined #freedesktop
ximion has joined #freedesktop
Rainer_Bielefeld has quit [Remote host closed the connection]
<bentiss>
robclark: FWIW, I got to ban another "Dakkshesh" just now :/
immibis has quit [Remote host closed the connection]
<bentiss>
emersion, __tim: I just restarted the last "large-*" machine we have, I'm going to do the same for the "server-*" ones, which means probably some connectivity issue
<__tim>
Thanks a lot!
<emersion>
ack!
<bentiss>
hopefully this will be enough to reset all of the network tables
<ifreund>
seems to be back online :)
<bentiss>
ifreund: I still have 2 machines to reboot
<bentiss>
and I suspect the one I rebooted was taking much of the heat, because I am one server down and it still seems OK
manuels39 has joined #freedesktop
<bentiss>
and now on to the last server
manuels3 has quit [Remote host closed the connection]
agd5f has joined #freedesktop
<bentiss>
OK, everything rebooted, hopefully there will be no more errors (ping me if that is the case)
<ifreund>
thanks!
pohly has quit []
Rainer_Bielefeld has joined #freedesktop
<bentiss>
I just banned one more user. But I wonder, what I I redirect tmate.io to a non working address on the runners, maybe that will be sufficient for them not being able to connect
immibis has joined #freedesktop
<bentiss>
for testing I need to stop one runner
jstein has joined #freedesktop
* bentiss
is testing if the counter-measure is working
Rainer_Bielefeld has quit [Remote host closed the connection]
<bentiss>
tomeu, __tim: can you add 'extra_hosts = ["ssh.tmate.io:192.0.2.1"]' to your [runners.docker] config, this will prevent the script kiddies to gaining ssh access to our servers
<bentiss>
tested with the equinix runners and it's working
<__tim>
ok, thanks
<bentiss>
maybe alatiera0 too ^^
<bentiss>
hopefully they will not start their own tmate server
Rainer_Bielefeld has joined #freedesktop
ybogdano has joined #freedesktop
anholt has quit [Read error: Connection reset by peer]
<__tim>
done
anholt has joined #freedesktop
Rainer_Bielefeld has quit [Remote host closed the connection]
jarthur has joined #freedesktop
___nick___ has quit []
___nick___ has joined #freedesktop
___nick___ has quit []
___nick___ has joined #freedesktop
Rainer_Bielefeld has joined #freedesktop
kem has quit [Ping timeout: 480 seconds]
kem has joined #freedesktop
ybogdano has quit [Ping timeout: 480 seconds]
<Lyude>
Hey daniels, bentiss did either of you start to get a bunch of gitlab account request emails?
<Lyude>
also y'all uh, missed the opportunity to respond to that person who wanted to do admin stuff
Rainer_Bielefeld has quit [Remote host closed the connection]
ybogdano has joined #freedesktop
<bentiss>
Lyude: sorry my fault. I fought against script kiddies the whole week end and this was one of the remediation
<bentiss>
I'll send a report to the board and the admins tomorrow
<bentiss>
the script is definitively borderline, but I fear it's a cover up for tmate (the name is kind of expressing this kind)
<bentiss>
oh, well, let's not take any chances, just banning that user
<anholt>
agree
<bentiss>
also, FWIW, I caught one attempt with my simple host override, so I guess this will do until they realise they can start up their own tmate/ssh server