daniels changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
columbarius has joined #freedesktop
co1umbarius has quit [Ping timeout: 480 seconds]
thaller is now known as Guest4292
thaller has joined #freedesktop
Guest4292 has quit [Ping timeout: 480 seconds]
ximion has quit [Quit: Detached from the Matrix]
tlwoerner has quit [Quit: Leaving]
tlwoerner has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
uajain has joined #freedesktop
<uajain>
Where do I contact gitlab.freedesktop.org admin staff?
<airlied>
here
AbleBacon has quit [Read error: Connection reset by peer]
bmodem has joined #freedesktop
Juest has quit [Ping timeout: 480 seconds]
<uajain>
airlied: Thanks, I want to request account deletion since I have lost all means of reviving it again
<airlied>
maybe just put the account name here and see if bentiss can look
GNUmoon has quit [Remote host closed the connection]
GNUmoon has joined #freedesktop
<bentiss>
uajain: happy to help (though it's an unusual request I must say). Would you mind changing your profile Skype (or LinkedIn or Twitter) name to "delete_me" so I'm sure the uajain here is the same uajain on gitlab.fd.o?
mvlad has joined #freedesktop
<bentiss>
FTR, (and the other admins) I fixed uajain account, so no deletion is required
<ishitatsuyuki>
I can't sign in. Says 'Could not authenticate you from GoogleOauth2 because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)". '
<emersion>
yeah, known issue
<bentiss>
ishitatsuyuki: yeah, known issue, see above (but I have a couple of more urgent ones first on my plate)
<ishitatsuyuki>
oh, missed that message. thanks
bmodem has quit [Ping timeout: 480 seconds]
<shadeslayer>
OOI have you tried using cloudflare to lower gitlab egress costs?
vkareh has joined #freedesktop
<pq>
I suppose not being able to log in makes the Gitlab frontend effectively read-only for everyone, so no further work can be lost in the mean time?
<emersion>
login works, external login doesn't
<emersion>
it would be a good idea to put everything in readonly, though
<mupuf>
bentiss: thoughts on the readonly proposal?
<emersion>
(if the issue can't be figured out, it would be nice to roll back to the old cluster, and continue the investigation on the new one)
<emersion>
(let me know if there's anything i can do to help -- but this is all dark magic to me…)
psukys has quit [Ping timeout: 480 seconds]
<bentiss>
emersion: the problem of rolling back is that we are going to lose git data since today's backup, which is why I wanted to do it as close as possible from the last backup fetch
<emersion>
you mean git data since we switched to the new cluster?
<emersion>
or more than that?
<bentiss>
git data since this morning (backup was on the old cluster), and it also requires 24h of downtime
<bentiss>
I've restored mesa
<bentiss>
FWIW
<emersion>
but 1 week worth of issues, comments, MRs, git, are lost
<bentiss>
1 week???
<emersion>
maybe a bit less
<__tim>
gstreamer has the same issue (and my private forks)
<bentiss>
emersion: you are talking about your libicc project?
<emersion>
i'm talking about all projects
<karolherbst>
I'm seeing comments from 3 hours ago
<bentiss>
from what I can see the db is fine, the git data not so much
<karolherbst>
so it's not all projects
<emersion>
karolherbst: but are you seeing comments from yesterday?
<karolherbst>
the most recently updated branch before the migration was pushed "Oct 18, 2023 7:10pm GMT+0200"
Inline has joined #freedesktop
ManMower has joined #freedesktop
<bentiss>
that would match when I did the copy of the image
<bentiss>
the initial one
<kxkamil>
igt-gpu-tools is back to Oct 19th
<bentiss>
yeah, Thu
<daniels>
karolherbst: that's telling you that you have a branch called rusticl/zink, so you can't create one called rusticl/zink/fixes/mr_1
<karolherbst>
mhhhhhhhhhh
<daniels>
nothing to do with gitlab, that's just git itself
<karolherbst>
right.. but I've deleted that one
<daniels>
seemingly not :P
<karolherbst>
hey, rewriting history is against the rules
<karolherbst>
*the past
<karolherbst>
but that would explain why those two branches are kinda lost
<karolherbst>
mhh well.. not really. just that git was reset to some older state I guess
<karolherbst>
bentiss: anyway yeah, I think this is just my fork having an old state and I have to delete some old refs
<karolherbst>
if you are okay with it, I remove the old ref, push the two new ones and see if that fixes my MRs, unless you want to repair more forks or something
<bentiss>
karolherbst: if you can, then yes please
<bentiss>
karolherbst: I can also fetch the backup data for you if you want
<karolherbst>
nah, that's all I have been working on anyway
<karolherbst>
okay, so that seems to have worked
<karolherbst>
and the MRs are fine as well
<karolherbst>
there was one odd error from git though:
<karolherbst>
maybe that error happened because there is an MR referencing that? no idea :)
<kxkamil>
should I wait for some time for restore?
<mupuf>
pushed 16h ago, and 2 days ago
<karolherbst>
yeah.. so it looks like git pushes in the past ~3 days are kinda lost
<mupuf>
*some* pushes
<karolherbst>
yeah, fair
<mupuf>
bentiss: is it possible to re-synchronize from the old server? I guess not...
<kxkamil>
or rather - when it will be safe to push?
<bentiss>
mupuf: yeah, I managed to restore mesa/main thanks to marge I think, we can consider anything that was pushed in the past couple of days "lost" as in everybody should re-push
<karolherbst>
slap an admin message onto gitlab and tell devs to repush their branches if they find broken MRs 🙃
<mupuf>
+1 to that
swatish2 has joined #freedesktop
<karolherbst>
if some git branches is the only thing lost I think that's fine
<bentiss>
mupuf: that's the problem, once I switched, I couldn't manage to make both copies up at the same time, I had to disable the one on the old server
<karolherbst>
I highlyt doubt there is anybody who wouldn't have the work locally anymore
<bentiss>
and disabling deletes the data :(
<mupuf>
bentiss: Right, I saw your sad smiley
<mupuf>
well, then so be it
<mupuf>
#goodenough
<mupuf>
and next time, we'll know that we'll need to do a cold migration
<bentiss>
I can also cherry pick in the backups, and I'm going to do that for emersion and other repos created in the past few days
<bentiss>
they are a little bit more of an issue because the repo doesn't exists on gitaly, and so it fails, really hard
<emersion>
bentiss: don't bother, i can re-create
<karolherbst>
any ETA on enabling external auth? Becuase that's the only way I can login.. otherwise somebody really should comment on MR 25848 so the contributor doesn't do something silly
<bentiss>
mupuf: this was a couple of hours of migration compared to a coupleof week or a month of monitoring and hand picking repos... I'll take the "too bad, it failed for the past 2 days
<bentiss>
emersion: k, I'll try to work on the others
<emersion>
ty for your work bentiss
<mupuf>
bentiss: right, fair enough!
<emersion>
next time, i should maybe try to watch over your shoulder so that i can learn
<mupuf>
thanks a lot, datacenter migrations suck!
<emersion>
if you'd be open to that
<bentiss>
mupuf: but next time I'll 1. put down gitlab, 2. do the snapshot migration, 3. change the main
<bentiss>
(without counting on mirroring actually happening)
<bentiss>
emersion: sure (hopefully not soon)
<mupuf>
I guess all we can do is pre-cache data in the hope of speeding up migration, but indeed, we probably should just have gitlab down while this is happening
<bentiss>
mupuf: yeah, pages is not in a good shape
<bentiss>
on both cluster
<bentiss>
I was trying to debug that before I got notified about the missing git history
<kxkamil>
I got Friday changes and few moments ago git tells I am ahead of remote ...
<karolherbst>
yeah, assume all changes to your fork from the last 3-4 days are gone
<kxkamil>
any hints? should I wait for a day?
<karolherbst>
just repush
<bentiss>
kxkamil: gitp ush -f
<mupuf>
bentiss: I'm writing a message to say the migration is done but that some branches/projects were lost in the process
<mupuf>
I;ll ask people to re-push, and if they can't contacting us to look for it in our backups
<mupuf>
sounds good? I'll send the message here when I am ready
<bentiss>
mupuf: sounds good
<bentiss>
thanks
<vyivel>
my avatar seems to be gone
<kxkamil>
ok, is it safe to push -f now?
<kxkamil>
better ti be safe ...
<karolherbst>
I already done it on my repo and it fixed the MRs I have opened against mesa. So maybe you should try one and see if it works and report back if something breaks
ibaranov has quit [Remote host closed the connection]
<ndufresne>
easy to say when you have handful changes, but having to do that on behavl of few hundreds submitters, that a lot of work
<karolherbst>
tldr: data migration didn't went well, git history lost for 3-4 days
ibaranov has joined #freedesktop
<karolherbst>
yeah.. if it's a lot of work, then ask bentiss once the message is out
<karolherbst>
but it only seem to affect pushes from the last 3-4 days
<vyivel>
https://gitlab.freedesktop.org/vyivel/glhf (which is a fork) is empty, trying to fork the original project again leads to the same link; is it safe to delete it and re-fork?
<bentiss>
karolherbst: what's teh omniauth error BTW
<karolherbst>
https cert stuff
<karolherbst>
Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)".
<bentiss>
huh. don't know which cert is an issue here :)
<emersion>
ah, the "no repository" page is actually only displayed when the Git data is missing, but the repo exists in the DB
<bentiss>
emersion: yeah
<bentiss>
so only for the past couple of days of new repos, I should be able to do that by hand
<karolherbst>
bentiss: maybe ours, because pages also causes https issues
<karolherbst>
*runs into
<bentiss>
karolherbst: pages is slightly different
<karolherbst>
oh noes
<bentiss>
some pages are pointing at the new cluster, some still to the old
<karolherbst>
ahh
<karolherbst>
tried to open the "callback" url for the oauth stuff?
<bentiss>
can't remember where to find that :)
<karolherbst>
:)
<mupuf>
"The migration is now over... but some git branches and new projects were lost in the process! Feel free to force push your local version to fix the merge requests. If your entire repo disapeared, please contact us on OTFC's #freedesktop channel."
<kxkamil>
pipelines cannot work, err 503
<bentiss>
mupuf: sounds good, thanks!
<mupuf>
ok, it's live!
<mupuf>
let's refine it as time goes
<vyivel>
typo in "disapeared"
<mupuf>
Thanks, will fix
<vyivel>
also vyivel/glhf is one of the repos that is gone
<mupuf>
vyivel: fixed
<bentiss>
ooh... *maybe* omniauth is related to IPv6
<ndufresne>
Same here, "Error: authenticating creds for "registry.freedesktop.org": pinging container registry registry.freedesktop.org: invalid status code from registry 503 (Service Unavailable)"
<vyivel>
"This project was scheduled for deletion, but failed with the following message: Failed to open TCP connection to gitlab-prod-registry.gitlab.svc:5000 (Connection refused - connect(2) for "gitlab-prod-registry.gitlab.svc" port 5000) "
<vyivel>
which i presume is the same error as the message above
<bentiss>
yeah, saw that registry is broken. Give me a minute
<ndufresne>
you can have more if you need ;-P (specially if it means a longer holding fix)
bmodem has quit [Ping timeout: 480 seconds]
emusial has joined #freedesktop
alexander has joined #freedesktop
<alexander>
Hello, is there a ETA on when third-party logins will work again?
<bentiss>
alexander: no, not yet, not sure what's the issue
<bentiss>
and I need to grab something to eat, bbiab
<alexander>
Ok, thanks! Its just that my mesa MR is now properly in need of a rebase, and I cant do it atm since my SSH key for my laptop is not registered :meme:
<Venemo>
hey guys
<Venemo>
it looks like gitlab absolutely destroyed my MR
<ndufresne>
thanks bentiss, worked for me too
<Venemo>
it seems like all (?) branches in my private repo were lost
<slomo>
bentiss: do you know who has access to the equinix runners? maybe they have a commit in their cache that apparently nobody else has :)
<slomo>
i mean, i have the change... but if i inject that myself the commit id changes and that might make someone unhappy
<Venemo>
are the contents of the branches actually lost or is it just the MR UI that is confused?
<slomo>
Venemo: lost, yes. if it's your MR, just push it again
<Venemo>
next time can we get some advance warning about possibly losing our work? I imagine some people might have had some work in branches in a private repo which are not backed up elsewhere
<mupuf>
Venemo: there are backups, just not for the past 2 days
<alexander>
Also, somewhat unrelated, but when is it acceptable to bump a MR? Im not sure how long I should wait before mentioning it, especially since its quite simple...
<Venemo>
mupuf: I guess it is fortunate that I haven't done too much in the past 2 days then, but still...
<mupuf>
Atomic updates with TB of data is hard. It was supposed to be transparent but it failed
<slomo>
bentiss: nevermind, we found it. ironically on a github mirror of the repo
<mupuf>
And knowing it failed required switching to the new data required removing the old one
<mupuf>
It is a shitty situation, but migrating TBs of data between multiple datacenters is slooooooow
<vsyrjala>
Venemo: you don't have that private repo as a remote in your local git repo?
<mupuf>
Took a month for the first sync
<Venemo>
vsyrjala: yes I have it, which is why I'm saying I'm fortunate
<emersion>
if you think you lost data, it can probably be recovered from the backup
<Venemo>
fortunately, it seems I didn't lose anything important, just had one MR messed up I think
<Venemo>
or maybe more than one, I would have to check
<Venemo>
yeah thus far 3 borked MRs
<Venemo>
also gitlab seems to be super slow now
<Venemo>
actually, 5 borked MRs
<Venemo>
I'm honestly not sure what to say. I don't understand why somebody feels that it's OK to mess up other people's work in this manner
<bentiss>
Venemo: I can't let you say that there were no warning. I put an un-dismissable banner saying that a migration will happen today. And I would have expected people to realize that it might come at a cost
<emersion>
Venemo: do you think people are infailible?
<emersion>
complaining is not constructive
<bentiss>
and now the situation is shit: only a couple of days is lost, which means that restoring everything from the last backup will take ~8-12 hours, for a little gain
<MrCooper>
Venemo: what makes you think anybody considers it OK?
<bentiss>
Venemo: and regarding "I don't understand why somebody feels that it's OK to mess up other people's work in this manner", the admin club is not closed. If you think you can do better, please join
<Venemo>
well, sorry but should I apologize to you for you deleting my branches?
<MrCooper>
no, just for your inappropriate attitude
<Venemo>
why is it inappropriate to ask questions here?
<Consolatis>
Venemo: this is not the right time for this. A proper post mortem of the whole situation would be interesting though (once things settle down)
<bentiss>
Consolatis: thanks. And yeah, if I spend time to speak here, I'm not restoring the service
<MrCooper>
this isn't how one "just asks questions"
<emersion>
it is inappropriate to make personal attacks like this
<Venemo>
I don't think I spoke to anyone here personally, I just came here to understand what happened
<Venemo>
I do remember seeing a notice about a migration, but it didn't mention that it will delete the contents from my MRs, hence I didn't expect that to happen, is all
<emersion>
nobody knew that this would happen
<Venemo>
now I come here to ask what's up and everyone's at my throat for that
<Venemo>
weird
<MrCooper>
you didn't even bother reading scrollback before you started complaining
<emersion>
you're aggressive, and you're wondering why people don't like your attitude?
<Venemo>
I'm not sure how to respond to that, so maybe better if I just don't
<Consolatis>
Venemo: Suggest just accepting things as they are for now. Complaining doesn't make it better, it just steals time which could be spend to fix things instead. In a few days when everything is settled these kind of questions can be discussed way more relaxed from all sides.
yann-kaelig has joined #freedesktop
<Venemo>
understood
<yann-kaelig>
Hello
<yann-kaelig>
Trying to reset my password on https://gitlab.freedesktop.org I get a "500 We're sorry. Something went wrong on our end."
gawin has joined #freedesktop
<gawin>
hi, nouveau wiki seems offline, not sure if due migration or something else earlier.
<bentiss>
gawin: yeah, gitlab pages is broken, looking into it right now
<bentiss>
yann-kaelig: yeah, external authentication is broken
<gawin>
Thanks, nothing important on my side, just noticed.
<yann-kaelig>
ok, thanks
<yann-kaelig>
Only for my curiosity. The migration will fix the spam gitlab issue ?
<emersion>
no
<yann-kaelig>
good
<yann-kaelig>
^^
<yann-kaelig>
Is it an issue that can be fixed in gitlab ?
blatant has quit [Quit: WeeChat 4.1.0]
<mupuf>
yann-kaelig: maybe, but never fully.
<mupuf>
bentiss: after discussing with Venemo, I'm proposing to update the banner to: "The migration is now over... but some git branches and new projects were lost in the process! If at all possible, force-push your branches to restore broken merge requests, or branches. If your entire repo disappeared, or you can't find the HEAD of a branch, please notify us on OTFC's #freedesktop channel and we'll try to address it as soon as possible. Sorry for the
<sima>
that seems to be the solution for missing git data
<sima>
you might need to delete some older branches if you have a name conflict
<alexander>
Its my first MR and branch so I dont think that will be a problem
<alexander>
Thanks!
<pobrn>
hi, I cannot log in with github: >>Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)".<< is this issue known?
<emersion>
yes
<__tim>
I found up to date commits in /srv/anongit.freedesktop.org/projects/ on annarchy btw, in case it helps anyone
<yann-kaelig>
I was using github to sign-in to gitlab.freedesktop.org. I
<yann-kaelig>
I'm not sure I created an gitlab.freedesktop.org and I deleted my github account. Can I recover my user used in gitlab.freedesktop.org ?
<yann-kaelig>
The thing is that I can not register if I want with the same username.
<pobrn>
yann-kaelig: have you tried "forgotten password"?
<yann-kaelig>
yes and I get a "500 We're sorry. Something went wrong on our end."
<pobrn>
yann-kaelig: well, me too when I want to log in, so I'm guessing there are some issues with gitlab atm still
<yann-kaelig>
ok, wil lsee but I'm curious right now, if now username has been registered and I can not use the same username, it's look like something could be broken with external authentification.
<yann-kaelig>
now/no
<daniels>
yeah, external authentication is broken, and will be fixed at some point, but right now there is no ETA
<pobrn>
daniels: I can't even log in using non-external auth
<yann-kaelig>
my bad, maybe I was not clear. I used github to authenticate on gitlab, but my github account has been deleted. Can I recover the username ?
<yann-kaelig>
on gitlab
<yann-kaelig>
or it's lost and I will have to create a new one
<yann-kaelig>
This is a deletion on github on my own initiative since it requires double authentication asking for my phone number
<yann-kaelig>
Not going to give my phon number for an account only used to report issues.
dmorin has joined #freedesktop
alexander has quit [Ping timeout: 480 seconds]
<bentiss>
yann-kaelig: I can quickly disable teh external auth for you (in theory)
<bentiss>
yann-kaelig: the link to github is deleted, maybe try re-asking a password
<Consolatis>
yann-kaelig: although kind of off-topic: GH doesn't require a phone number. You can use any random TOTP application (on your desktop or phone) for the 2nd factor.
<bentiss>
FWIW, gitlab-pages should be fixed except for www.networkmanager.dev. Not sure why the HTTP01 challenges fails
<dmorin>
I'm also experimenting authentication failure using GoogleOauth2. Error reported is certificate verify failed (hostname mismatch)
<bentiss>
dmorin: yeah same issue with GH or gitlab.com, still unsure what is happening
<bentiss>
our certs are correct in theory ;/
AnuthaDev has joined #freedesktop
<pobrn>
bentiss: any idea why I wouldn't be able to log in using local credentials? my account was created through github, but I set a password on fdo gitlab proper a couple days ago
<yann-kaelig>
Consolatis: Well, that was not explained or maybe not clear enough and hidden somewhere, but I remembe, however, that the news announced a mandatory double authentication linked to a phone number
<bentiss>
pobrn: I don't know, no, sorry
<yann-kaelig>
bentiss: still the same issue.
electrodeyt has joined #freedesktop
<bentiss>
I think I found out why we were having 500 errors -> I had too many webservice pods on the old cluster
<bentiss>
could also be something wrong with ml-29
AnuthaDev has quit []
gawin has quit [Ping timeout: 480 seconds]
mripard_ has joined #freedesktop
mripard has quit [Ping timeout: 480 seconds]
<bentiss>
now the registry is fixed...
Mario_ has joined #freedesktop
<Mario_>
Hi; I've noticed that I can't authenticate using Github anymore to gitlab.freedesktop.org.
<Mario_>
Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.54:443 state=error: certificate verify failed (hostname mismatch)".
<emersion>
known issue
<Mario_>
OK thanks
<MrCooper>
not using external auth, so I can log in, keep getting logged out though FWIW
<bentiss>
MrCooper: are you using a consistent IP or is it flickering?
<bentiss>
MrCooper: if the IPv4 points at the old cluster when IPv6 is available on the new, chances are that the redis cache is not happy and logs you out
gawin has joined #freedesktop
<MrCooper>
you mean the IP used by Firefox? Not sure
anholt has joined #freedesktop
gawin has quit []
<bentiss>
MrCooper: works here. When you say "keep getting logged out", is like you can not login or is it after a couple of minutes?
<bentiss>
emersion: to be able to restore libicc, did you just git push -f, or have you done something on the UI too?
<emersion>
bentiss: "init empty repo" in the UI, then push
<bentiss>
emersion: k, thanks, much easier than me initializing the repo in the gitaly pod ;) thanks!
<MrCooper>
bentiss: after some time
nanokatze has joined #freedesktop
<nanokatze>
hey, when trying to log into my fdo gitlab account with my gitlab.com account, I get "Ssl connect returned=1 errno=0 peeraddr=18.154.227.54:443 state=error: sslv3 alert handshake failure", is this known?
<emersion>
known issue
<nanokatze>
thanks
nanokatze has quit [Quit: Page closed]
<Consolatis>
might be worth adding to the banner (or a new banner) that there are currently issues with 3rd party logins and that its being worked on
<bentiss>
mupuf: ^
<bentiss>
Consolatis: good point!
AbleBacon has joined #freedesktop
ximion has joined #freedesktop
<bentiss>
alright. Manually restored all recently created repository to a working state (to their upstream counterpart), So now, we should only ask people to re-push their changes
<bentiss>
OK... so if I use the old cluster IP address, oauth works.
<bentiss>
now I "just" need to compare the 2 kubernetes states :/
<anholt>
is "some git pushes appear to have been lost in the git trees but the web ui knows they happened" known?
<bentiss>
anholt: yes, the db is fine, but the git data no. So just re-pushing the git tree is sufficient to restore teh MRs and whatnot
bl4ckb0ne has joined #freedesktop
<koike>
is "yaml invalid Undefined error (01HDERKVBND9T3VHHMY3492SJQ) " known ? I pushed the same commit hash to two branches, one of them was valid and the other invalid
MrBonkers has joined #freedesktop
paulosv has joined #freedesktop
<paulosv>
Hi, after the gitlab migration, I am having some issues logging in and registering... registering gives 500 after submiting the form, and logging in via Gitlab.com throws an error "Could not authenticate you from GitLab because 'Ssl connect returned=1 errno=0 peeraddr=18.154.227.126:443 state=error: sslv3 alert handshake failure".' Are you aware of this / where can I report this?
<emersion>
yeah, that's a known issue
<paulosv>
ok, thx
Inline has quit [Remote host closed the connection]
<paulosv>
and do you know whether at least Gmail login works now? :D
<bentiss>
hmm... "Error during Akismet spam check, flagging as not spam: SSL_connect returned=1" that seems to be the same
paulosv has quit [Remote host closed the connection]
yann-kaelig has quit []
<bentiss>
Haha! running on a webservice pod "curl https://google.com" returns an SSL error
<emersion>
bad CA store?
<bentiss>
and the same works on the old cluster
<bentiss>
so yes, bad CA store.... but why???
<bentiss>
emersion: do you know off hand how to fix this? it seems /etc/ssl/certs is correctly populated on the pod
<bentiss>
on the pod: dpkg -l ca-certificates -> ii ca-certificates 20210119 all Common CA certificates
<bentiss>
so it's installed
swatish2 has quit [Ping timeout: 480 seconds]
<bentiss>
Ahhh, on the old cluster /etc/pki/ca-trust/extracted/pem is filled, on the new, not
<daniels>
bentiss: iirc it’s update-ca-certificates you want
<daniels>
at the very least, try tab-completing from update-
ylatuya[m] has quit [Ping timeout: 480 seconds]
razze[m] has quit [Ping timeout: 480 seconds]
LaughingMan[m] has quit [Ping timeout: 480 seconds]
adziahel[m] has quit [Ping timeout: 480 seconds]
Nova[m] has quit [Ping timeout: 480 seconds]
nirbheek_ has quit [Ping timeout: 480 seconds]
Wallbraker has quit [Ping timeout: 480 seconds]
dabrain34[m]1 has quit [Ping timeout: 480 seconds]
chrysn[m]1 has quit [Ping timeout: 480 seconds]
ttancos[m] has quit [Ping timeout: 480 seconds]
seaweed[m] has quit [Ping timeout: 480 seconds]
swick[m] has quit [Ping timeout: 480 seconds]
mitTengiz[m] has quit [Ping timeout: 480 seconds]
Hazematman has quit [Ping timeout: 480 seconds]
msizanoen[m] has quit [Ping timeout: 480 seconds]
unrznbl[m] has quit [Ping timeout: 480 seconds]
Pope_Rigby[m] has quit [Ping timeout: 480 seconds]
gnfzdz[m] has quit [Ping timeout: 485 seconds]
gkiagia has quit [Ping timeout: 480 seconds]
nazarewk[m] has quit [Ping timeout: 480 seconds]
cassidy[m] has quit [Ping timeout: 480 seconds]
pitch[m] has quit [Ping timeout: 480 seconds]
jtatz[m] has quit [Ping timeout: 486 seconds]
cmeissl[m] has quit [Ping timeout: 486 seconds]
gallo[m] has quit [Ping timeout: 481 seconds]
Sumera[m] has quit [Ping timeout: 481 seconds]
jenatali has quit [Ping timeout: 480 seconds]
TillMaas[m] has quit [Ping timeout: 480 seconds]
bendlas[m] has quit [Ping timeout: 480 seconds]
pac85[m] has quit [Ping timeout: 480 seconds]
siddh has quit [Ping timeout: 480 seconds]
zredshift[m] has quit [Ping timeout: 480 seconds]
nee[m] has quit [Ping timeout: 480 seconds]
leigh_2valid[m] has quit [Ping timeout: 480 seconds]
BLumia[m] has quit [Ping timeout: 480 seconds]
ewlsh[m] has quit [Ping timeout: 480 seconds]
dabrain34[m] has quit [Ping timeout: 481 seconds]
Sid127 has quit [Ping timeout: 480 seconds]
twopubsolar[m] has quit [Ping timeout: 481 seconds]
koike has quit [Ping timeout: 480 seconds]
tintou has quit [Ping timeout: 480 seconds]
kusma has quit [Ping timeout: 480 seconds]
tomeu has quit [Ping timeout: 481 seconds]
aenderboy[m] has quit [Ping timeout: 480 seconds]
sergi1 has quit [Ping timeout: 480 seconds]
M0xMRTTOld[m] has quit [Ping timeout: 480 seconds]
heftig has quit [Ping timeout: 480 seconds]
marcel203s[m] has quit [Ping timeout: 480 seconds]
nielsdg has quit [Ping timeout: 480 seconds]
tinywrkb has quit [Ping timeout: 480 seconds]
havdan[m] has quit [Ping timeout: 480 seconds]
Trevinho[m] has quit [Ping timeout: 480 seconds]
alatiera[m] has quit [Ping timeout: 481 seconds]
dcbaker has quit [Ping timeout: 480 seconds]
Leopold__ has quit []
Leopold_ has joined #freedesktop
Kayden has quit [Quit: -> JF]
Mario_ has quit [Remote host closed the connection]
linyaa has joined #freedesktop
<bentiss>
daniels: thanks... but /etc/pki is mounted as ro, and this is populated in an init container
heftig has joined #freedesktop
alatiera[m] has joined #freedesktop
<bentiss>
even weirder. I killed on pod on the old cluster, and the ca store is fine but /etc/pki/ca-trust/extracted/pem/ is empty too
<bentiss>
oops. I thought I was in a pod when I looked at /etc/pki... but I was on my own computer. At least that explains why ;)
Kayden has joined #freedesktop
M0xMRTTOld[m] has joined #freedesktop
<bentiss>
so... the problem is I'm getting redirected to cloudfront instead of contacting google.com in my curl experience
<bentiss>
probably a coredns config issue then
phire has quit [Remote host closed the connection]
phire has joined #freedesktop
<bentiss>
maybe a quick fix for the night is to restore the old IP address
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
<bentiss>
alright, this is done. And it seems to be working
<bentiss>
hanging up for today
<Consolatis>
bentiss: thanks for your work the whole day
LaughingMan[m] has joined #freedesktop
Leopold___ has joined #freedesktop
damian has quit []
ylatuya[m] has joined #freedesktop
paulosv has joined #freedesktop
Leopold_ has quit [Ping timeout: 480 seconds]
Leopold___ has quit []
paulosv has quit [Remote host closed the connection]
vkareh has quit [Quit: WeeChat 4.0.5]
Leopold_ has joined #freedesktop
Leopold_ has quit []
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
MrBonkers has joined #freedesktop
MrBonkers has quit []
MrBonkers has joined #freedesktop
Leopold has joined #freedesktop
_whitelogger has joined #freedesktop
MrBonkers has quit [Quit: Ping timeout (120 seconds)]
sima has quit [Ping timeout: 480 seconds]
MrBonkers has joined #freedesktop
Leopold has quit []
ngcortes has joined #freedesktop
<ngcortes>
anybody else having an issue with gitlab where it can't find a source branch for an MR?