daniels changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
afanyulionel has joined #freedesktop
<afanyulionel> Hello everyone, my name is Afanyu Lionel. I'm very excited to join the Debian community and contribute. My goal is to help improve Debian while also learning from all of you. If you have any tips for a new contributor, please let me know! I'm eager to get involved and make positive changes. Thanks.
afanyulionel has quit [Remote host closed the connection]
jarthur has quit [Ping timeout: 480 seconds]
pkira has quit [Ping timeout: 480 seconds]
co1umbarius has joined #freedesktop
columbarius has quit [Ping timeout: 480 seconds]
AbleBacon has quit [Read error: Connection reset by peer]
DodoGTA has quit [Quit: DodoGTA]
DodoGTA has joined #freedesktop
todi1 has quit [Ping timeout: 480 seconds]
sima has joined #freedesktop
Leopold_ has quit [Remote host closed the connection]
todi has joined #freedesktop
Leopold_ has joined #freedesktop
todi1 has joined #freedesktop
todi has quit [Ping timeout: 480 seconds]
a-l-e has joined #freedesktop
a-l-e has quit [Remote host closed the connection]
a-l-e has joined #freedesktop
todi1 has quit [Ping timeout: 480 seconds]
a-l-e has quit [Ping timeout: 480 seconds]
psykose has quit [Remote host closed the connection]
psykose has joined #freedesktop
tales-aparecida has quit [Remote host closed the connection]
mairacanal has quit [Remote host closed the connection]
An0num0us has joined #freedesktop
pjakobsson has quit []
An0num0us has quit [Ping timeout: 480 seconds]
<bentiss> gitlab security update incoming in the next few minutes
<bentiss> (and so upgrade to 16.4.x)
<bentiss> all gitaly pods have restarted, db migration is now done, and the webservice pods are rolling to the new version
mvlad has joined #freedesktop
<bentiss> and we are now on gitlab 16.4.1
<mupuf> bentiss: thank you!!
<mupuf> released yesterday, that was fasr!
<cassidy[m]> Is the update still going on? I now get git@gitlab.freedesktop.org: Permission denied (publickey) when trying to pull or push
<cassidy[m]> now I can push to my personal repo but I still get this error: remote: GitLab: Internal API unreachable
<mupuf> bentiss: ^
<mupuf> I can reproduce
<mupuf> pushes fail
<mupuf> it is weird, I can ssh into git@gitlab.freedesktop.org, but git push fails saying my key isn't valid
ximion has quit [Quit: Detached from the Matrix]
<mupuf> hmm, one push went through, so maybe a gitlaly server needs rebooting?
<mupuf> daniels, bentiss: ^?
<bentiss> From the pods themselves, I don't see an error, digging into the logs
<mupuf> thanks :)
<jani> AOL!
<bentiss> mupuf: yeah, maybe we'll have to reboot the nodes. The restart of k3s from yesterday maybe didn't fix everything
<bentiss> (because right now I can not extract logs from loki, which is unusual)
<mupuf> bentiss: :s
<mupuf> jani: America OnLine?
<jani> mupuf: yeah, ancient stuff
<jani> "Me, too!"
<mupuf> was that in an advertisement?
<mupuf> I see :D Too young for this ;)
<jani> yeah :)
<mupuf> I mean, I could technically have been experiencing this, but it would have required France to be more anglophone and have better internet availability
* bentiss really wonders why generating the initrd is taking forever (doing an upgrade of the nodes while on it)
emusial has joined #freedesktop
todi has joined #freedesktop
<emusial> Hi, is freedesktop down?
<bentiss> emusial: somehow, yes, maybe
<emusial> :(
<bentiss> the webservice pods are all having issues...
<bentiss> and I'm seeing some weird kernel errors
<bentiss> the db is probably not in a good shape
<bentiss> once server-6 reboots, I'll migrate the db there, so it doesn't play whack a mole
todi has quit [Ping timeout: 480 seconds]
<emusial> keeping my fingers crossed
<bentiss> rebooting once server-6 seems to have done something
<bentiss> but I interrupted apt upgrade, so I have to reboot it once more
<emusial> :D
<Venemo> is gitlab down for everyone or just me?
<bentiss> Venemo: everyone, yes
<Venemo> :(
<bentiss> Venemo: got to reboot all the nodes, so it'll roughly take 4 hours (at most, I think)
<Venemo> whoah.
<emusial> :o
<pq> Recharge day! \o/
<bentiss> pq: yep, expect that this was unexpected
<bentiss> this gets slowly back on its feet
<bentiss> (which means one node out of six rebooted)
<emusial> bentiss: gitlab was very stable for years, what is happening? I mean, I do not remember outages like this in past and now it is not first time in last weeks
<bentiss> emusial: the cluster is getting old, and I need to do some upgrades on to it
<emusial> makes sense...
<bentiss> emusial: last time I attempted a migration on a new cluster, but it failed, so I reverted back to the old one
<mceier> gitlab security update happened; now it's very secure ;)
<bentiss> mceier: yes, security by preventing anybody to enter
<emusial> :D
pkira has joined #freedesktop
<Venemo> I suppose I should set up a repo on github or something, to use for emergencies like this
<emusial> we did opposite... we started using gitlab everywhere instead github because it was more stable and now... suprise, it stopped to be :D
<bentiss> Venemo: not telling you what to do, but please wiegh in the number of downtime with respect to maintaining a backup repo, without CI
<Venemo> I didn't mean that as an insult or anything
<bentiss> emusial: just to be clear, we never gave a guarantee on the uptime. It's all best effort
<bentiss> Venemo: I know, I didn't took it as such
bmodem has joined #freedesktop
<Venemo> my immediate problem is that I want to continue my work on a different computer and as gitlab is down I have nowhere else to push what I'm working on now
<bentiss> FWIW, I'm rebooting a second server now, and ceph is holding a lot of locks on the kenerl, which probably explains why it stopped again
<emersion> Venemo: you can add a remote with your other computer (via SSH)
<bentiss> Venemo: if you have a local pull, you can always configure it as a remote on your other computer
<emusial> bentiss: noone can give us quaranee, it is just observation that it was more stable in past and it was our taken risk to start using it in such scope ;)
<Venemo> hmm, that actually is a good idea
<Venemo> thanks for the suggestion :)
<bentiss> that's the whole point of git: it's distributed :)
<bentiss> no surprises it's not working well: https://paste.centos.org/view/54b05b72 2 nodes are not ready when they should be ready
<bentiss> mupuf: rebooting everything was indeed needed
<mupuf> bentiss: good!
todi has joined #freedesktop
todi has quit [Read error: Connection reset by peer]
swatish2 has joined #freedesktop
todi has joined #freedesktop
suja has joined #freedesktop
<bentiss> 4 servers rebooted, 2 to go, but I need to wait for the data to settle first
todi has quit [Ping timeout: 480 seconds]
<bentiss> hmm... I think ceph stops working when I reboot because of the nodown flag I have to set otherwise the OSD are killing each other
dos1 has quit [Ping timeout: 480 seconds]
dos1 has joined #freedesktop
todi has joined #freedesktop
bozo16 has quit [Remote host closed the connection]
bozo16 has joined #freedesktop
<bentiss> OK, there is only one server to reboot now
<mupuf> crossing fingers
todi1 has joined #freedesktop
todi has quit [Ping timeout: 480 seconds]
columbarius has joined #freedesktop
<bentiss> that shutdown was a bliss compared to the others... no ceph network errors :)
co1umbarius has quit [Ping timeout: 480 seconds]
todi1 has quit [Ping timeout: 480 seconds]
<bentiss> server rebooted, still a couple of disks not up yet
<bentiss> \o/ ceph seems to be good now
<eric_engestrom> thanks bentiss!
<bentiss> please tell me if you are still experiencing issues
<mupuf> bentiss: thanks!!!
mairacanal has joined #freedesktop
<mupuf> did you hear back from packet wrt the new cluster?
<bentiss> mupuf: yes. They told me that I could take my time :)
<bentiss> so I'm continuing to work on it as time permits
<mupuf> Good!@
<bentiss> yep
suja has quit [Quit: Page closed]
bmodem has quit [Ping timeout: 480 seconds]
mripard has joined #freedesktop
thaller has joined #freedesktop
<zmike> is creating MRs broken?
vkareh has joined #freedesktop
<eric_engestrom> zmike: it was, a few hours ago, but it should be fixed now
<eric_engestrom> what error are you getting?
<zmike> 404
<pq> zmike, are you sure you're logged in?
<pq> I think the reboots today logged everyone ouit
<mupuf> pq: I get the same 404
<mupuf> zmike: did you make this link by hand?
<zmike> no
<zmike> from git
<zmike> oh
<zmike> I got logged out
<zmike> :/
<mupuf> hehe
<pq> I would hope I wouldn't have the permission to create a MR from someone else's fork into an upstream repo arbitrarily :-P
<eric_engestrom> yeah, I don't like the way all the services are now returning 404 when it should be 403, just to avoid divulging the existence of the resource
<eric_engestrom> I get it, but I don't like it
<zmike> it's nonobvious
<eric_engestrom> exactly
<zmike> if I wake up and the first page I load in my browser is firing off a MR then I have no way to know
An0num0us has joined #freedesktop
<psykose> the ole security and usability being mutual exclusives
a-l-e has joined #freedesktop
a-l-e has quit [Ping timeout: 480 seconds]
thaller has quit [Read error: Connection reset by peer]
cat has joined #freedesktop
Thymo has quit [Ping timeout: 480 seconds]
Junie has joined #freedesktop
<Junie> Hello everyone, My name is Isoken Ibizugbe and I am an outreachy applicant. I look forward to working on open source software with the linux community.
<mupuf> Junie: welcome!
Leopold__ has joined #freedesktop
GNUmoon2 has quit [Remote host closed the connection]
GNUmoon2 has joined #freedesktop
Leopold_ has quit [Ping timeout: 480 seconds]
Junie has quit [Remote host closed the connection]
a-l-e has joined #freedesktop
Thymo has joined #freedesktop
a-l-e has quit [Ping timeout: 480 seconds]
ximion has joined #freedesktop
co1umbarius has joined #freedesktop
columbarius has quit [Ping timeout: 480 seconds]
psukys has joined #freedesktop
a-l-e has joined #freedesktop
alanc has quit [Remote host closed the connection]
alanc has joined #freedesktop
An0num0us has quit [Ping timeout: 480 seconds]
a-l-e has quit [Ping timeout: 480 seconds]
flom84 has joined #freedesktop
An0num0us has joined #freedesktop
flom84 has quit [Ping timeout: 480 seconds]
a-l-e has joined #freedesktop
jarthur has joined #freedesktop
An0num0us has quit [Ping timeout: 480 seconds]
<bwidawsk> Anyone know offhand projects on gitlab that have an alpine container with a custom kernel for me to copy from?
<alatiera> speaking of templates
<alatiera> I was thinking of adding a "blob" template, where it has no install/update/etc actions and you need to specify the base_image
<alatiera> idea being for image based things without package managers
heapify has joined #freedesktop
AbleBacon has joined #freedesktop
heapify has quit []
nektro_ has quit [Remote host closed the connection]
nektro_ has joined #freedesktop
flom84 has joined #freedesktop
flom84 has quit []
jarthur has quit [Read error: Connection reset by peer]
jarthur has joined #freedesktop
mvlad has quit [Remote host closed the connection]
damian has joined #freedesktop
Haaninjo has joined #freedesktop
a-l-e has quit [Ping timeout: 480 seconds]
functionguyy has joined #freedesktop
a-l-e has joined #freedesktop
<functionguyy> Dear Linux Community members, I'm an Outreachy applicant for the December 2023 cohort. I'm happy to be here
functionguyy has quit [Remote host closed the connection]
Kayden has quit [Quit: Leaving]
Kayden has joined #freedesktop
K`den has joined #freedesktop
Kayden has quit [Read error: Connection reset by peer]
K`den is now known as Kayden
vkareh has quit [Quit: WeeChat 4.0.4]
psukys has quit [Ping timeout: 480 seconds]
swatish2 has quit [Ping timeout: 480 seconds]
sima has quit [Ping timeout: 480 seconds]
a-l-e has quit [Ping timeout: 480 seconds]
An0num0us has joined #freedesktop
damian has quit []
functionguyy has joined #freedesktop
An0num0us has quit [Quit: WeeChat 3.5]
functionguyy has quit [Remote host closed the connection]
Leopold__ has quit [Remote host closed the connection]
Leopold_ has joined #freedesktop
iNKa has joined #freedesktop
Brocker has quit [Read error: Connection reset by peer]
glehmann has quit [Remote host closed the connection]
Venemo has quit [Remote host closed the connection]
Haaninjo has quit [Quit: Ex-Chat]
karolherbst has quit [Ping timeout: 480 seconds]
nnm has quit []
nnm has joined #freedesktop
afanyulionel has joined #freedesktop
afanyulionel has quit [Remote host closed the connection]