ChanServ changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
K`den has joined #freedesktop
Kayden is now known as Guest1084
K`den is now known as Kayden
Guest1084 has quit [Ping timeout: 480 seconds]
ybogdano has quit [Remote host closed the connection]
ybogdano has joined #freedesktop
Seirdy has quit []
Seirdy has joined #freedesktop
Seirdy has quit [Ping timeout: 480 seconds]
Seirdy has joined #freedesktop
ybogdano has quit [Ping timeout: 480 seconds]
danvet has quit [Ping timeout: 480 seconds]
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
Seirdy has quit [Ping timeout: 480 seconds]
Seirdy has joined #freedesktop
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
bleb has joined #freedesktop
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
Seirdy has quit [Ping timeout: 480 seconds]
Seirdy has joined #freedesktop
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
ngcortes has quit [Remote host closed the connection]
alatiera has quit [Read error: Connection reset by peer]
alatiera has joined #freedesktop
jarthur has quit [Ping timeout: 480 seconds]
jarthur has joined #freedesktop
alatiera has quit [Read error: Connection reset by peer]
alatiera has joined #freedesktop
ximion has quit []
alatiera has quit [Quit: Ping timeout (120 seconds)]
alatiera has joined #freedesktop
enilflah has joined #freedesktop
jarthur has quit [Remote host closed the connection]
holmanb_ has quit [Remote host closed the connection]
holmanb has joined #freedesktop
sumits has joined #freedesktop
Brocker has quit [Read error: Connection reset by peer]
Brocker has joined #freedesktop
sadlerap has quit [Ping timeout: 480 seconds]
GNUmoon has quit [Ping timeout: 480 seconds]
sadlerap has joined #freedesktop
GNUmoon has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
GNUmoon has quit [Quit: Leaving]
GNUmoon has joined #freedesktop
danvet has joined #freedesktop
mvlad has joined #freedesktop
vbenes has joined #freedesktop
MajorBiscuit has joined #freedesktop
___nick___ has joined #freedesktop
___nick___ has quit []
___nick___ has joined #freedesktop
lfiedoro has joined #freedesktop
pjakobsson_ has quit []
pjakobsson has joined #freedesktop
<emersion>
okay, so picking up where i left off yesterday
<emersion>
i'm looking at a helmfile diff output, ran on the config repo with no modifications
<emersion>
i see some +++ and ---, why are there changes when i didn't make any?
<daniels>
the shared-secrets job gets rebuilt every time, unnecessarily
<daniels>
the Docker registry secret is also shown as spuriously changing but that’s fine
<daniels>
oh and the migrations job gets recreated as well, because the Helm part doesn’t know if it’ll be needed or not
<daniels>
if it’s just those three changes then you’re fine
alatiera has quit [Ping timeout: 480 seconds]
alatiera has joined #freedesktop
<emersion>
sounds good then
<emersion>
so, to perform the upgrade, would the following be enough:
<emersion>
- check out the branch auto-created by helm bot
<emersion>
- helmfile diff
<emersion>
- helmfile -i apply
<emersion>
- merge MR
<emersion>
?
<bentiss>
except that you need to set the environment to packet-HA
<emersion>
yeah `--environment packet-HA` ofc
<bentiss>
in theory, you can just squash the 2 steps diff and -i apply: helmfile -e packet-HA -l chart=gitlab -i apply -> and make sure to not type 'enter' while reviewing the changes
<bentiss>
also, the bot should give in the MR the rough changes for what to expect, so you can go slightly faster after looking at the MR
<emersion>
what are the things to look out for when reviewing the diff?
<bentiss>
basically if there are just changes related to gitlab version you are fine
<emersion>
ok
<bentiss>
if there are some other changes, like labels applied to pod, or config changes, you need to check where that comes from
<bentiss>
usually the release notes are enough to know if there is some bigger impact (like the updated nginx last time)
<emersion>
let's give this a try then
<emersion>
hm that big diff overflows my terminal buffer
<emersion>
diff looks good, applying now
<emersion>
oops.
<emersion>
phew
<emersion>
got a temporary 502, but everything back to normal now
<emersion>
\o/
<emersion>
hmmm
<emersion>
the admin UI is still showing 14.6.2 for some reason
<bentiss>
yeah, 502 and lag in the admin panel are expected:
<daniels>
emersion: watch -n5 kubectl --context fdo -n gitlab get pod -l 'app==webservice'
<bentiss>
502 means that nginx was trying to relay the request to a webservice pod that was just killed
<daniels>
the update is rolling, so you'll see a pattern of new-version webservice pods being created and starting up, then the old ones shutting down
<bentiss>
and the lag is because you are still served by an old webservice pod (see daniels comment)
<emersion>
daniels: "Init" means still pending upgrade?
<emersion>
and "Running" all green?
<daniels>
emersion: you'll see Pending (not assigned to a node), then ContainerCreating (pulling the new image), Init (per-container-instance setup), then eventually it becomes Running
<bentiss>
emersion: Init means that it is waiting on a condition in the init containers to be good before starting the pod
<emersion>
cool!
<daniels>
if the pods aren't converging to Running, you can use kubectl --context fdo -n gitlab describe pod/${podname}
<daniels>
that'll show you the steps in more details (the two init containers it needs to run first, then what's happening with the webservice+gitlab-workhorse containers)
<daniels>
oh, and remove --context fdo - that's not necessary since you only have one k8s context which is marked as the default - I made it so I explicitly have to type out --context fdo to bash on prod
<bentiss>
FLHerne: see above, we just did an upgrade
<bentiss>
emersion: one last thing about upgrades, for small upgrades like that we can do them anytime, but it's better to make them in the European morning, when .au is fading its work, and US hasn't started
<emersion>
ack
<bentiss>
for security fixes, depending on the security, we can do it whenever we want
<emersion>
thanks for guiding me through the whole process!
<bentiss>
for upgrades that require a migration 14.6 ->14.7), it's even better to do it on a Saturday or Sunday, because the migration can take up to 1h
<emersion>
i see
<bentiss>
but in overall, people are not complaining too much when we upgrade :)
<bentiss>
emersion: anyway, thank you a lot and wewlcome to the team as a kubernetes master :)
<emersion>
:)
<FLHerne>
Thanks, missed that it was still going on
karolkro2 has joined #freedesktop
karolkro1 has quit [Read error: Connection reset by peer]
karolkro2 has quit [Remote host closed the connection]
karolkro2 has joined #freedesktop
<daniels>
emersion: thanks, glad to see it all went through ok!
lfiedoro has quit [Remote host closed the connection]
Haaninjo has joined #freedesktop
jarthur has joined #freedesktop
ybogdano has joined #freedesktop
MajorBiscuit has quit [Ping timeout: 480 seconds]
MajorBiscuit has joined #freedesktop
ybogdano has quit [Ping timeout: 480 seconds]
MajorBiscuit has quit [Ping timeout: 480 seconds]
ybogdano has joined #freedesktop
vbenes has quit []
demarchi has quit [Ping timeout: 480 seconds]
demarchi has joined #freedesktop
captainc1ris has joined #freedesktop
<daniels>
~/wayland/weston weston-buffer-lifetime % kubectl --context fdo -n gitlab get pod
<daniels>
Error from server (ServiceUnavailable): the server is currently unable to handle the request
<daniels>
uh
<daniels>
bentiss: k3s and/or etcd seem very unhappy