<jcristau>
it looks like a proxad issue this time around. or between proxad and whoever's supposed to be next on that path.
<bentiss>
that seems terribly close to what we had yesterday (except I was getting 503s)
<jcristau>
not really no
<bentiss>
well, I mean yesterday it was fastly that was proxying the traffic that was down, when today, we got "something" outside of proxad which is down and which seems to not proxy the traffic to github.com
<bentiss>
anyway, I found the way to work around that deployment problem, so I'll stop complaining :)
<daniels>
bentiss: wfm ...
lumpi0 has joined #freedesktop
<daniels>
bentiss: also thanks for yesterday's updates! that's awesome
lumpi0 has quit [autokilled: Suspected spammer. Mail support@oftc.net with questions (2021-06-09 09:11:58)]
<bentiss>
daniels: no worries
<bentiss>
daniels: works now
<bentiss>
daniels: if you could in the next days (week?) migrate marge bot to the new cluster, that would be good
<daniels>
bentiss: oh yeah, that's easily done, there's no persistent state so it should just be moving the env in the helmfile
<daniels>
bentiss: is everything in git up to date?
<bentiss>
daniels: marge is on the config repo, correct?
<daniels>
bentiss: yep, in gitlab-bots/
<daniels>
bentiss: should I change the environment name there to packet-HA?
<bentiss>
daniels: omnibus is up to date, and I just pushed one more change on config. There are a few changes in the vector and network-policy, but that shouldn't affect gitlab-bots
<daniels>
got it :)
<bentiss>
daniels: if you want, that might be easier in the future, yes
<bentiss>
daniels: not sure if you saw, but I deployed velero for backing up and transfering workloads across clusters
<bentiss>
daniels: it might be an otpion for migrating the db if I can not get slony (hot-replication db) to work
<daniels>
bentiss: every time we talk I end up having to google a new tool ...
<daniels>
bentiss: that sounds fine, or tbh the database is still small enough that we could do it during weekend downtime? I'm happy to do that again
<bentiss>
the thing is with slony we could do a downtime of a few minutes, so it won't have to be on Sunday
<bentiss>
with velero, I think one hour should be enough too, because snapshotting the disk took 5-7 min, so restoring the raw data should not take that many more time
<bentiss>
with the "normal" backup/restore we are down to at least 4 hours, like last time
pupskuchen has joined #freedesktop
pupskuchen has quit [Remote host closed the connection]
<bentiss>
velero would be easy I think, I already did it a few times for grafana, indico, and monitoring
<bentiss>
slony.... well, I need to build up a container, and do changes in the db first, so that's risky
chomwitt has quit [Ping timeout: 480 seconds]
ladyluna has joined #freedesktop
ladyluna has quit [Remote host closed the connection]
revolve has joined #freedesktop
revolve has quit [Remote host closed the connection]
<bentiss>
daniels: would this Sunday morning be OK for your to do the db migration?
<bentiss>
I should be able to do it by myself, but having a second pair of hands would be great
<daniels>
hmm, I'd thought it was a lot less than 4h, but maybe I'm just suppressing it ;)
<daniels>
sure, Sunday morning is fine and I can try to do some reading on Velero before
<bentiss>
daniels: cool, thanks, I'll set up a banner
<bentiss>
velero is a lot of fun: for monitoring, all I had to do was
<bentiss>
on the old cluster: velero backup create monitoring-2021-06-08-11-27 --include-namespaces monitoring --default-volumes-to-restic
<bentiss>
on the new cluster: velero restore create --from-backup monitoring-2021-06-08-11-27
<bentiss>
there are a few subtleties if the workload is already present, but for postgres and redis, all we have to do is delete the deployments and pvc, run velero restore
<bentiss>
daniels: what you can do too, is try migrating marge-bots with velero
<bentiss>
that would give you a first hand test
<bentiss>
daniels: and also, server-2 has mc setup for accessing velero bucket: "mc ls velero"
wesleyac has joined #freedesktop
wesleyac has quit [autokilled: Suspected spammer. Mail support@oftc.net with questions (2021-06-09 10:26:04)]
jdost has joined #freedesktop
jdost has quit [Remote host closed the connection]
mceier has quit [Ping timeout: 480 seconds]
mceier has joined #freedesktop
<daniels>
oh wow, that's super cool
<daniels>
marge doesn't have any PV though :P
sandroandrade has joined #freedesktop
sandroandrade has quit [Remote host closed the connection]
<bentiss>
well, velero is about the kubernetes workloads/secrets/configs/etc... and restic is about the PVs
<bentiss>
so I still think it'll be good for you to try
<daniels>
oh ok, gotcha :)
V has quit []
V has joined #freedesktop
mceier has quit [Remote host closed the connection]
mceier has joined #freedesktop
ximion has joined #freedesktop
liyang has joined #freedesktop
liyang has quit [Remote host closed the connection]
chomwitt has joined #freedesktop
mykiwi has joined #freedesktop
mykiwi has quit [autokilled: Suspected spammer. Mail support@oftc.net with questions (2021-06-09 13:48:09)]
Cerise has joined #freedesktop
Cerise has quit [Remote host closed the connection]
mceier has quit [Remote host closed the connection]
adjtm has left #freedesktop [Leaving]
mceier has joined #freedesktop
pjdc has joined #freedesktop
pjdc has quit [Remote host closed the connection]
ximion has quit []
imirkin is now known as Guest1453
scubasteve has joined #freedesktop
scubasteve has quit [Remote host closed the connection]
blue__penquin has quit [Remote host closed the connection]
bittin[m]2 has joined #freedesktop
bittin[m]2 has quit [Remote host closed the connection]
rak-zero has joined #freedesktop
rak-zero has quit [Remote host closed the connection]
scastano has joined #freedesktop
scastano has quit [Remote host closed the connection]
ximion has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
chomwitt has quit [Ping timeout: 480 seconds]
jrayhawk has quit []
jrayhawk has joined #freedesktop
jrayhawk has quit []
jrayhawk has joined #freedesktop
chomwitt has joined #freedesktop
jrayhawk has quit []
jrayhawk has joined #freedesktop
ngcortes has joined #freedesktop
aic1 has joined #freedesktop
aic1 has quit [Remote host closed the connection]
ngcortes has quit [Remote host closed the connection]
jarthur has joined #freedesktop
AnatolyTroyan-t has joined #freedesktop
AnatolyTroyan-t has quit [Read error: Connection reset by peer]
luna_ has joined #freedesktop
luna_ has quit [Remote host closed the connection]
ngcortes has joined #freedesktop
reillybrogan has quit [Ping timeout: 480 seconds]
Daenks has joined #freedesktop
Daenks has quit [Remote host closed the connection]
geonic17 has joined #freedesktop
geonic17 has quit [autokilled: spambot. Dont mail support@oftc.net with questions (2021-06-09 20:42:51)]
meiskam has joined #freedesktop
meiskam has quit [autokilled: Suspected spammer. Mail support@oftc.net with questions (2021-06-09 20:51:18)]
chomwitt has quit [Ping timeout: 480 seconds]
danvet has quit [Ping timeout: 480 seconds]
neunon has joined #freedesktop
neunon is now known as Guest1497
Guest1497 has quit [Remote host closed the connection]
nickre-M has joined #freedesktop
nickre-M has quit [Remote host closed the connection]