03:35 <Kayden> friend pointed me at https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/

04:09 <mupuf> DragoonAethis, robclark: indeed, this is what we did before... but we could do it with kubernetes now rather than hosting services on individual servers

04:10 <mupuf> that being said, this would once again be a lot of work

04:10 <mupuf> and would require local people

07:35 <bentiss> mupuf, DragoonAethis, robclark: yeah lots of effort plus having a S3 service externally managed would save us a lot of admin time, so hosting this in a uni might not be the best choice.

07:37 <bentiss> that being said, Hertzner has a coloc option where we could fit 14 units for $111 per month + electricity

08:10 <ivyl> bentiss: BTW thanks for helping with estimating the costs and all the work you do on the infra <3

08:11 <ivyl> yeah, I'm a fan of trading some extra costs in favor of our admin time. The less we have to admin ourselves and instead use managed stuff, as long as the costs are sensible, the better :-)

08:11 <ivyl> so that's the angle I'm exploring

08:15 <mupuf> Ditto!

08:16 <mupuf> GitLab provides enough infra work, no need to add extra

09:05 <bentiss> ivyl: do you mind if I tweak a little bit your numbers you sent on the email and put them in the gitlab issue? (and also thanks for that initial quote)

09:23 <ivyl> go ahead

09:28 <bentiss> thx

09:40 <tintou> Hi there, I'm getting remote: remote: fatal: unable to write loose object file: No space left on device when git pushing to a repository (not that everything still seems to work though)

09:56 <bentiss> tintou: which repo please?

09:56 <tintou> virglrenderer

09:56 <bentiss> thx

09:57 <bentiss> we still have disk space available on the gitaly pods...

10:02 <bentiss> could be that kemper is out of disk

10:02 <bentiss> yep... kemper is out of disk, so the post-receive hook is failing

10:03 <bentiss> Mithrandir, daniels ^^

10:19 <daniels> fixed, thanks

10:36 <bentiss> thx

12:51 <bentiss> (I know they have the free credits, but still)

12:52 <mupuf> oh, and if we go with hetzner, I would vote for moving the infra out of the USA

12:53 <bentiss> yeah, last treasurer report said we started the year with $230000, so that's 4 years of Hetzner if we just burn cash without any income

12:54 <mupuf> London or Amsterdam would be great locations IMO

12:54 <pinchartl> mupuf: +1

12:55 <mupuf> B. Less political uncertainty

12:55 <mupuf> A. Most of our DUTs are at Collabora which IIRC means cambridge

12:57 <bentiss> I thought Hetzner we had the choice between Germany or Finland....

12:58 <ivyl> yep, Hetzner is either or one of those

12:58 <bentiss> well, they have USA and Singapore... but what's teh point of goign there :)

12:59 <ivyl> extra latency!

12:59 <xorgy> being near a large proportion of contributors, and the most interconnected backhaul in the world...

13:00 <mupuf> I see, then it would be germany then

13:00 <xorgy> (not Singapore lol)

13:00 <mupuf> for OVH, it could be london or AMS

13:01 <xorgy> Hetzner has for me always given superior performance to other European hosts, whether virtual and dedicated

13:03 <xorgy> OVH has been really challenging to work with, in my experience. Billing errors are a common problem I, and people I know, run into with them.

13:10 <pq> Someone repeatedly cutting those comms cables in the Baltic Sea might make people think twice of hosting things in Finland, sadly.

13:12 <mupuf> pq: hehe, yeah!

13:17 <svuorela> I'm hoping that when/if fundraising is needed that we could trick downstream consumers like distributions and KDE/Gnome/... to also help at least boost & convey the messages...

13:18 <mupuf> svuorela: I would think that XDC sponsorship would be enough to cover for CI expenses

13:19 <ivyl> it might be, and we have people on the board working on securing donors that would explicitly help with the infra costs

13:22 <mupuf> <3

13:24 <svuorela> I btw showed the gitlab issue to the KDE gitlab people, and one of the comments from them was "yeah. I can point to that when people ask why I'm so hard on purging CI artifacts.."

13:29 <mupuf> hehe

13:31 <daniels> yeah, the Collabora machines are in Cambridge, so anything in western or central Europe is totally fine

13:32 <mupuf> The main valve farm is in Seattle... but I don't think it matters

13:34 <mupuf> given the current latency of gitlab, I don't think the network matters much. Hopefully improved S3 performance will more than compensate for the increased ping time

13:56 <bentiss> mupuf: my bet is currently on the db and ceph that kills our latency

13:57 <mupuf> the DB being on ceph is for sure not gonna win any race ;)

13:57 <bentiss> yeah. sorry for that bad choice

13:59 <mupuf> sorry? What else could we have done?

14:00 <mupuf> tying the DB to a single host would have made migrations even harder

14:00 <bentiss> have the db replicated at the db level, and each db talk to its own ssd (like what I think we should do if we migrate to Hetzner)

14:01 <bentiss> basically postgres already has all the redundancy in place (with a pgbouncer IIRC), so you don't need ceph to replicate the db 3 times, you just need 3 dbs

14:09 <mupuf> oh, yeah, that's better

14:22 <__tim> GStreamer runners are all on Hetzner as well, but, as you say, runners are probably the least problematic piece of all of this

15:13 <DragoonAethis> mupuf: I tested our smaller CBL Postgres on a beefy enough Ceph vs local NVMe, and the penalty was ~10x in throughput and ~25x latency

15:13 <DragoonAethis> This is on 100G NICs and PCIe 4.0 NVMes

15:14 <DragoonAethis> So +1 for moving to local storage, just keep in mind Hetzner's block storage is AFAIK also networked and not local - benchmark it as needed

15:17 <bentiss> DragoonAethis: thanks for the info. Honestly it can not be worse than what we have. And if we have to pay for 2 more bare metal servers at $66, that's probably something that we could afford (we can apparently put on the same network bare metal and vcpus)

15:20 <DragoonAethis> bentiss: it's not gonna be worse, just benchmark as needed :P It's also billed per hour, so quickly starting a test project and tearing it down a few hours later costs very little (holler if you want a test project with API access)

15:22 <DragoonAethis> If you're running on dedicated boxes, you can also request custom local NVMe drives to be installed (for a considerable price, but still cheaper than anything else you'll find in big clouds)

15:23 <bentiss> TBH, my problem with benchmarking the db is that I just don't know how to do it. I would be able to set up the db once/if I get Hetzner access, restore a backup onto those dbs, but bloody hell I have no ideas on how to test this without pulling the entire gitlab :)

15:25 <DragoonAethis> The way I did it was: Restore a backup onto the new server, monitor the production instance for longer queries, rerun these long queries a few times on the to warm caches and then rerun them with perf monitoring (exec time, EXPLAINs on production vs test setup) to figure out a good baseline

15:26 <bentiss> ... which is gibberish to me :)

15:27 <DragoonAethis> it doesn't get any easier when you're 20 tabs into EDB docs :)

15:27 <bentiss> (not that I don't understand all the terms, but I don't know how to apply them and probably don't want to)

15:28 <bentiss> and again, if anyone wants to help on the db side, I'll welcome any

16:53 <xorgy> pq: I think that issue w/ Finland may be resolved relatively soon, so I wouldn't factor it in too much if it makes sense to host in Finland for cost or other reasons.

17:29 <bentiss> damnit, I mixed up PSU and OSUOSL... :)

17:34 <bentiss> brain fart... if arm donates runners, could we configure them so that they are tied to mesa (easy) and only used for marge-bot (with some rules magic). This should allow control of the egress because more of the same jobs will be running and we wouldn't have to wipe the cache all the time, plus this means more trusted runners as they are not free for all

17:34 <daniels> yeah, we can manage that

17:35 <bentiss> I mean mesa and or drm or any trusted project

17:36 <bentiss> daniels: do you agree on taking the immediate Hetzner solution for gitlab? And when we are less in emergency see the various options?

17:36 <bentiss> (I'm already looking at ways on how to deploy the db, and it seems https://cloudnative-pg.io will be the best)

17:39 <daniels> bentiss: heh, I was just trying to compose a reply

17:39 <daniels> tbh I think AWS + Fastly is better for now

17:39 <daniels> in that I do think Hetzner if we have the time and bandwidth to manage it is great, but then we need to make sure we have a steady stream of cash coming in - which is not something we've been used to

17:40 <daniels> so rather than having to run around with cap in hand, personally my preference would be to give the board a year or two to figure out how to do the whole sponsorship thing, and make sure we can actually get enough money and reproduce that annually

17:40 <bentiss> k, I'll do with whatever we are given in the end :)

17:41 <daniels> it's also not my decision - I'm just a dude on the internet with some opinions

17:41 <bentiss> same here :)

17:41 <bentiss> anyway, going afk for tonight. Have fun everybody

17:45 <daniels> me too - enjoy!

17:57 <mupuf> Hmm, getting sponsorship with fastly and aws will take time, and xorg has enough cash to last us for multiple years, that's enough time to figure out what to do

17:58 <mupuf> I would maybe host some runners at AWS and everywhere we can get capacity for free, but keep the core of the infra on paid infra, for stability and reliability

17:59 <daniels> yeah, I can definitely see the argument

18:07 <mupuf> also, with infra we can rent by the hour, we can experiment more easily with new ways of doing things... unlike what happened with equinix where we had to take the infra down when doing major upgrade.

18:19 <DragoonAethis> Neat feature here: You can take a VM snapshot with a runner setup, pay all the time just for its storage (€0.01353/GB/month with 23% VAT) and quickly bring new runners up/down on demand from that

18:44 <mazze> Did something change in ftp.freedesktop.org? When I try to access it with FileZilla I get ECONNREFUSED error.

18:55 <bilboed> <xorgy> "OVH has been really challenging..." <- I'll up that. I've had (and sadly still have) servers on linode and OVH/scaleway before trying hetzner and it was day and night. People stress about the pricing difference, but I would focus on the "bang for the buck". Not only are they cheaper but they also offer *way* better cpu/iops. The only downside is... 1G only connectivity, which is a not really an issue if you switch to their (new)

18:55 <bilboed> object storage