01:37 <DemiMarie> emersion: yup, very cheap

01:38 <DemiMarie> if you want to make it real expensive you have to either make a legitimate user work more, do fingerprinting (ewwww), rely on IP addresses (limited effectiveness), or rely on some sort of cryptographic challenge

01:38 <DemiMarie> Are X.org membership email verification emails down?

06:18 <colinmarc> <bentiss> "pinchartl: don't know, but I..." <- the WAF thing should be, no?

06:18 <colinmarc> that's part of their marketing materials, at least

06:22 <colinmarc> <DemiMarie> "emersion: yup, very cheap" <- in theory they can afford it. in practice a little bit of friction goes a looong way here, I think.

06:27 <colinmarc> that said, I think a workable solution would be for big sites who get scraped the most to basically publish the lists of IPs that are scraping them automatically. then smaller sites can just set up an automatic ip ban for those lists.

06:28 <colinmarc> * them automatically on a regular basis. then

07:23 <pinchartl> colinmarc: the issue is that you'd end up blocking half of the internet. unlike search engine bots that rely on a "small" pool of (relatively) well-behaved hosts, AI scrappers use botnets with millions of devices

07:25 <pinchartl> and I wouldn't be surprised if those botnets were made of either compromised devices, or devices "borrowed" from unsuspecting owners (all kind of home appliances)

07:25 <colinmarc> are you talking about the big AI companies, or other parties?

07:26 <colinmarc> I would be surprised if openAI/meta/etc are using botnets. consumer devices is possible, but if they get their own consumer devices blocked, that's their problem, I think

07:27 <bilboed> The problem is that there's no accountability for being a bad actor on the Internet. If ISPs really told their clients "hey, you're a bad actor, another strike and you're banned" it might have some impact. Right now people don't even care that their devices/computers are hacked, it doesn't impact them

07:37 <bilboed> (Most Hosting providers do that. But I guess they have "less" clients, and the impact of having their AS banned/blacklisted is catastrophic)

11:41 <slomo> 503 "SSL handshake error" on gitlab currently

11:41 <bentiss> slomo: sorry, I'm trying to get the real ip of clients, and I broke the config

11:43 <slomo> no worries, just wanted to be sure you're aware of it :)

11:45 <bentiss> \o/ this seems to be working now (getting the actual IP, not the one from the fastly POP) -> this should help Anubis

11:47 <bentiss> but the problem is now the runners are talking directly to the hetzner LB, bypassing fastly and we don't have their IPs

12:01 <bentiss> reverted to the old configuration, because there were too many corner cases where we don't get the actual IP

13:29 <eric_engestrom> jenatali, alatiera: the windows runners seem to all be busy, not picking up marge jobs after almost an hour: https://gitlab.freedesktop.org/mesa/mesa/-/jobs/74461531

13:33 <alatiera> the two gst runners are alive

13:33 <alatiera> seems to be a slow day overall

13:33 <alatiera> I also have jobs queued for 50m atm

13:35 <eric_engestrom> ack

13:35 <eric_engestrom> job just got picked up

13:45 <__tim> jenatali, the Windows shared runners sponsored by Microsoft seems to have been offline for the last 2 days, is that known/expected?

13:47 <__tim> last contact was on Apr 9 at 3:33:33am apparently :D

14:51 <eric_engestrom> bentiss: more s3 issues... something's breaking cors when loading pngs from s3: https://mesa.pages.freedesktop.org/-/mesa/-/jobs/74434844/artifacts/results/summary/results/trace@gl-intel-amly@0ad@0ad-v2.trace.html

14:52 <eric_engestrom> > CORS header ‘Access-Control-Allow-Origin’ missing

14:55 <bentiss> eric_engestrom: any ideas what I should put in https://oxyno-zeta.github.io/s3-proxy/configuration/example/ ? (sorry I got my brain on something else, but I happily apply any change)

14:55 <eric_engestrom> I'm very rusty on cors things, but I think that header needs to be in the s3.freedesktop.org redirect response with the *.your-objectstorage.com domain

14:55 <eric_engestrom> looking at your link, give me a minute

14:56 <eric_engestrom> bentiss: can I see the current config?

14:56 <bentiss> https://gitlab.freedesktop.org/freedesktop/helm-gitlab-deployment/-/blob/main/configs/hetzner/s3-proxy-config.gotmpl?ref_type=heads

14:56 <eric_engestrom> my first guess would be `cors:allowOrigins:["fsn1.your-objectstorage.com"]`

14:59 <eric_engestrom> no, wait, I think I got it backwards, origin is the domain of the webpage the user has opened

14:59 <eric_engestrom> so it would be all the *.pages.freedesktop.org that exist... not really something we can enumerate

14:59 <eric_engestrom> if globs are allowed in there then that should do it

15:00 <eric_engestrom> or maybe we can jsut disable cors? I don't think other websites embedding resources from s3.freedesktop.org can actually cause any issues?

15:10 <eric_engestrom> after reading up a bit more, I feel fairly confident that the only thing that could be done if we disable cors is that someone could create a webpage that loads a url from s3 that is not public and requires auth by a specific few users (and then eg. uploads it elsewhere), and then trick one of those few users who have access to open that malicious web page, which will then use that user's creds to download that resource

15:12 <eric_engestrom> given the kind of contents we have on there, I don't think this is much of an issue, so I think we can indeed disable cors on s3.freedesktop.org

15:13 <eric_engestrom> I'd love to hear a second opinion from someone who knows more about cors and the kind of resources available through s3.fdo :)

15:25 <eric_engestrom> hmm, cors is disabled by default (https://oxyno-zeta.github.io/s3-proxy/configuration/structure/#servercorsconfig), so adding `server: cors: enabled: false` won't solve anything 😅

15:26 <eric_engestrom> > allowOrigins [...] This support stars in origins.

15:26 <eric_engestrom> so I think we need to enable it with `*.freedesktop.org` as the value; I'll post an MR

15:26 <daniels> I'm comfortable with disabling CORS, but otoh this doesn't apply to direct requests or HTTP redirects - it applies only to stuff which is fetched from JS

15:27 <daniels> so yeah, enabling CORS + allowOrigins *.fd.o seems like the right move

15:32 <bentiss> eric_engestrom, daniels: deploying eric_engestrom's MR ATM

15:32 <bentiss> eric_engestrom: it's deployed now

15:33 <bentiss> doesn't seem to change a bit in your link

15:33 <eric_engestrom> yeah, I also just tried

15:33 <eric_engestrom> my browser still says "CORS header ‘Access-Control-Allow-Origin’ missing"

15:34 <eric_engestrom> so that's not the right fix

15:34 <bentiss> but... isn't the fsn1 resource that should set the header? not s3-proxy?

15:35 <bentiss> if so... that's going to be a tight harder to set

15:38 <eric_engestrom> oh

15:38 <eric_engestrom> I think you're right

15:42 <eric_engestrom> bentiss: according to https://docs.hetzner.com/storage/object-storage/howto-protect-objects/cors/ it's disabled by default; do you know if you or another admin enabled it?

15:43 <bentiss> nope, nobody should have touched that

15:44 <bentiss> should be easy enough to set up

15:46 <eric_engestrom> but if it's disabled, then it allows everything right? so that shouldn't be where the issue is?

15:47 <bentiss> maybe it's on the other side the problem

15:47 <bentiss> I've just set the CORS rule on the s3 bucket

15:47 <bentiss> doesn't change a bit

15:47 <bentiss> sigh

15:49 <eric_engestrom> revert everything before the weekend (both my MR and what you just did), just in case it's breaking something else

15:49 <bentiss> eric_engestrom: point 6 at https://docs.hetzner.com/storage/object-storage/howto-protect-objects/cors/ -> it's the requester that needs to set the header

15:50 <bentiss> OK, both removed

15:52 <eric_engestrom> the requested sets the Origin header, but the server sets the Access-Control-Allow-Origin header that tells the browser which resources are allowed to read the response

16:00 <eric_engestrom> bentiss: added `Access-Control-Allow-Credentials: true` to the MR, although I'm not really clear whether it can help

16:01 <eric_engestrom> I'm still not sure at what layer the problem is, I might still be looking in the wrong place

16:02 <eric_engestrom> oh

16:03 <eric_engestrom> hold on, I think it might be a problem with the html we generate

16:03 <eric_engestrom> we set crossorigin="Anonymous"

16:03 <eric_engestrom> I think removing that might fix everything

16:04 <eric_engestrom> "By default (that is, when the attribute is not specified), CORS is not used at all." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Attributes/crossorigin)

16:05 <eric_engestrom> and if we do opt in, then the cors chain must be there (s3 must set the right headers, and hetzner must set the right headers)

16:08 <eric_engestrom> been here since https://gitlab.freedesktop.org/mesa/piglit/-/commit/99be1b06ff360edc17e9117bd82189dbade42013; tomeu do you know why you added that?

16:13 <eric_engestrom> hmm wait, I don't think that's the right file to look at, but it's the only thing that contains `crossorigin`, so I don't understand what else could be setting that

16:16 <eric_engestrom> fyi I'm off until thursday (except mesa release stuff on wednesday), so I won't be able to look into this issue more until then

16:17 <eric_engestrom> I have verified that removing `crossorigin="Anonymous"` from that page fixes the issue though, so if someone can figure out which file sets this, please remove it :)

16:19 <bentiss> cool

16:20 <bentiss> and FWIW, I'll be off next week too

16:21 <eric_engestrom> ack

16:22 <eric_engestrom> tomeu, daniels: posted https://gitlab.freedesktop.org/mesa/piglit/-/merge_requests/1005 which removes the one `crossorigin` that could find; if there's another one, please remove it too, and once they're all removed, please uprev piglit in mesa ci ❤️

16:23 <bentiss> OK... so I'm now bypassing the loadbalancer from fastly to directly talk to the nodes. This way, we will count the egress on each node AND we will have the proper IP addresses in the logs

16:50 <bentiss> and gitlab feels much more reactive now that we have 5gbs of bandwith instead :)

16:54 <bentiss> (and now anubis.gitlab.freedesktop.org works much better)

17:36 <bentiss> \o/ I didn't break drm: https://lore.kernel.org/all/174434262168.3947740.5175447386950205073.pr-tracker-bot@kernel.org/ the PR has been pulled by Linus ;)