02:32 <mupuf> eric_engestrom: thanks for looking into it! I saw this issue earlier, and what they actually want is to prioritize jobs within a pipeline (https://gitlab.com/gitlab-org/gitlab/-/issues/323100#note_632289328)

02:34 <mupuf> What I believe we want is not a collaborative approach where projects set priorities.... unless it required admin rights. I believe what we actually want is for runners themselves to say what they want to run preferentially

02:36 <mupuf> Or just be able to query the list of jobs they can pick, then try to grab a job by providing the list of jobs it wants to run, sorted by priority

02:37 <mupuf> This way, the runner is fully in control of what it runs... which sounds good to me!

03:00 <mupuf> eric_engestrom: oh, I see you commented on the issue

03:01 <mupuf> I'll comment on it tomorrow :)

03:01 <mupuf> thanks!

07:59 <__tim> I sometimes wonder if the way runners pick jobs couldn't be improved, but perhaps I misunderstood how it works. aiui if a runner has 6 idle slots, it may basically go and pick/schedule 6 jobs in one go, is that right? but now if we have 5 runners with 6 idle slots each that would mean that the first runner has picked all the jobs and is superbusy

07:59 <__tim> and the others are still idle, which seems suboptimal?

08:00 <__tim> unrelated to 'priorities' though I guess

08:02 <bentiss> one thing we can easily do, is have the jobs being conditional on marge, and when they are, tag the jobs with "marge". In each runner, we register 2 agents, one specifically for marge (with concurrent of 3) which can not run untagged jobs, and one for the rest (concurrent of 2), which can run marge or untagged jobs

08:02 <bentiss> this way we always reserve some capacity to marge

08:03 <bentiss> should be easy enough to do with extends and rules

10:17 <eric_engestrom> mupuf: ah, that issue has a lot of comments; I only skimmed them, and I missed that they reduced its scope to "priority of jobs within a pipeline" (instead of priority of jobs of different pipelines)

10:18 <eric_engestrom> bentiss: we could, but that would leave unused cycles when marge isn't busy

10:19 <bentiss> eric_engestrom: well, if we manage to redefine the tag "marge" into "upstream project" I am sure we could use that for any project

10:19 <eric_engestrom> I don't know if we have a graph of marge pipelines vs other pipelines throughout the day, but I kind of expect marge peaks at the end of the day while the other pipelines start peaking during the day

10:20 <eric_engestrom> (with "day" being blurry because there's a mix of timezones, but mostly EU & US so the blur is of 8 pixels^W hours)

10:21 <bentiss> we could also simply deploy one extra agent with concurrency 1 to reserve a part of the runners, a smaller part

10:22 <bentiss> but yeah, having global pipelines priorities would be simpler

12:38 <daniels> __tim: the request_concurrency runner config param defines how many jobs it'll pull at once

12:38 <daniels> __tim: we typically set that to 2 iirc, and then have a lower check_interval, so the odds are that jobs will be distributed 2 across each runner

12:48 <__tim> ah, interesting, thanks

12:48 <__tim> not sure we set that one

14:02 <bentiss> daniels: hey. Regarding migrating MinIO, I might have an even better solution based on Istio

14:03 <bentiss> daniels: Istio has the ability to check JWT token and refuse a request with an invalid token

14:03 <bentiss> daniels: Istio/envoy can also have lua script to do... whatever we want

14:05 <bentiss> daniels: so I am trying to see if I could not bypass entirely the ceph short tokens and rely only on jwt: if the request has an AWS token -> let ceph decide. If there is a JWT -> use OPA to validate, and if valid, add a server side static AWS token never seen by the caller. If no auth is provided, let opa decide and add the token on the server on success

14:06 <bentiss> daniels: this should basically avoid (maybe) keycloak but also now only a simple curl would be enough to push/get files from ceph, without having to authenticate

14:16 <vsyrjala> finally it occurred to me to click the 'Receive notifications about your own activity' button to allow me to search through my own gitlab comments by searching through my mailbox