alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
thelounge90 has quit []
thelounge90 has joined #freedesktop
thelounge90 is now known as alatiera
xexaxo_ has joined #freedesktop
aleksander has joined #freedesktop
V has quit [Ping timeout: 480 seconds]
V has joined #freedesktop
<daniels>
it's going to drop out for a few minutes now, in the service of attempting to make it better ...
<daniels>
bentiss: ^ they're all still connection-refused to redis, even though redis is happily running the entire time and logging no errors; I've bumped the core socket connections-waiting limit higher
<bentiss>
daniels: ok cool
<bentiss>
daniels: I wonder if the network issues are not wireguard related
<bentiss>
cause here and now we just lose connectivity between the pods
V has quit [Ping timeout: 480 seconds]
<daniels>
bentiss: oh yeah, that would probably do it too ...
<daniels>
is that the kilo svc dying, or?
<bentiss>
more likely the flannel wireguard backend
<bentiss>
no ideas on how to switch to a different backend though :(
<daniels>
could we create a VPC in Packet and just route everything directly over that rather than using WG?
<bentiss>
" but if you continue to run into problems after rebooting all the nodes with the new flag set, you're probably best off rebuilding the cluster."
<daniels>
...
<bentiss>
daniels: next question then is: do we need to encrypt the traffic between the pods? knowing that the utlimate goal is to have the runners handled there too
<daniels>
bentiss: hmmm, I would vote for having the runners in a separate cluster + VPC, given that they don't need to have access to anything other than public endpoints
<daniels>
in which case, no they don't need to be connected, as long as the main cluster has its own separated VPC rather than public
<bentiss>
yep, the full hybrid will not be available
<bentiss>
:(
<bentiss>
though witht the redundancy we have, it shouldn't be an issue if packet does switch maintainance
<daniels>
nod
<daniels>
plus if it means we can avoid WG randomly dropping out (how did you diagnose that?) then it'll be more reliable on the whole anyway :P
<bentiss>
re wireguard: I see in the logs some connections that never reach their destination
<bentiss>
for example, in the "Loki stack monitoring" dashboard, there are a bunch of "error: write tcp 10.40.7.114:3101->10.40.0.7:46238: write: broken pipe"
<bentiss>
also, if you look at the osd logs when they are failing, they complain about connectivity issue
<bentiss>
I am not 100% sure wg is the one to blame, but this is definitively one piece in the middle that is potentially a breaker
<bentiss>
not to mention that getting rid of it will enhance the disks operations on ceph
<bentiss>
daniels: anyway, I have way too much things to do these weeks: some internal work + my presentation for XDC. I don't think I'll have time to play with layer 2. Feel free to start playing with it
pendingchaos has joined #freedesktop
xexaxo_ has joined #freedesktop
<daniels>
oh interesting! I’m in the same boat tbh, will be at least a couple of weeks till I can start doing something meaningful :\
<bentiss>
I guess I need to add that to the slides and preach for help :)
<daniels>
++++++++++
MrCooper has quit [Quit: Leaving]
MrCooper has joined #freedesktop
ximion has joined #freedesktop
aleksander has quit [Quit: Leaving]
bcarvalho has quit [Ping timeout: 480 seconds]
daniels has quit []
daniels has joined #freedesktop
jstultz has quit []
jstultz has joined #freedesktop
bcarvalho has joined #freedesktop
bcarvalho has quit [Remote host closed the connection]
agd5f_ has joined #freedesktop
agd5f has quit [Read error: Connection reset by peer]
xexaxo_ has quit [Ping timeout: 480 seconds]
xexaxo_ has joined #freedesktop
Haaninjo has joined #freedesktop
tchar has quit []
tchar has joined #freedesktop
thaller has quit [Remote host closed the connection]
xexaxo_ has quit [Ping timeout: 480 seconds]
thaller has joined #freedesktop
xexaxo_ has joined #freedesktop
xexaxo_ has quit [Ping timeout: 480 seconds]
bcarvalho has joined #freedesktop
jstein has joined #freedesktop
zhxuxu_ has joined #freedesktop
zhxuxu has quit [Ping timeout: 480 seconds]
zhxuxu_ is now known as zhxuxu
zhxuxu has quit [Quit: Leaving]
zhxuxu has joined #freedesktop
jarthur has joined #freedesktop
ngcortes has joined #freedesktop
tomeu has quit [Quit: Ping timeout (120 seconds)]
tomeu has joined #freedesktop
danvet has quit [Ping timeout: 480 seconds]
jjardon[m] has joined #freedesktop
immibis has joined #freedesktop
jstein has quit []
ngcortes has quit [Remote host closed the connection]