daniels changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
<bentiss>
yeah migration did not happen correctly, so I have to restart it, it'll take a couple of hours (maybe?)
<eric_engestrom>
bentiss: what are the parts that are not working right now?
<eric_engestrom>
(I'm getting random errors when loading comments or diffs or pipelines)
<__tim>
ah, does this explain why pipelines aren't being created for pushes?
<eric_engestrom>
yeah, I assume it's all related
mvlad has joined #freedesktop
<bentiss>
sorry, I was in a meeting
<bentiss>
long story short, the migration took 60 min, got killed and some parts of gitlab is therefore not updated
<eric_engestrom>
ack; I assume waiting for it to run is all we can do in the meantime? do you know which parts are affected, and which parts are safe to use?
<eric_engestrom>
eg. pipelines look like they're not working, but perhaps we can still review MRs for instance?
<bentiss>
eric_engestrom: no idea. The db is in a middle state between 16.8 and 16.9, so it's still safe to use, but I have no idea what is broken
<eric_engestrom>
ack
<eric_engestrom>
thanks :)
<bentiss>
sorry. The past few migrations took a few minutes, but this one clearly was bigger (and I have no idea on how long it'll take before running it)
<eric_engestrom>
it's ok, don't worry :)
<bentiss>
just re-running it took a few secs, and the pods are now restarting
<eric_engestrom>
perhaps a banner can be added on gitlab to inform users of this, so that they don't get annoyed at seemingly random things not working?
<bentiss>
it'll be over in a couple of minutes
<eric_engestrom>
ah ok
<eric_engestrom>
nvm then :P
<bentiss>
2 more pods to restart, but most of the workload should be handled by the new version
<bentiss>
regarding the pipeline, there is a high chance that the sidekiq pods were crashing, but are in a good shape now
<bentiss>
everything should be back to normal now
vx has quit [Quit: G-Line: User has been permanently banned from this network.]
vx has joined #freedesktop
<bentiss>
I've retried all of failed gitlab background jobs in the past 6 hours, so the pending MR/pipelines should now be created
<bentiss>
welcome to gitlab 16.9.1
<eric_engestrom>
awesome, thanks bentiss!
<eric_engestrom>
so there shouldn't be any manual retry needed by anyone, eg. pushing a new commit to re-create pipelines, and things like that?
<bentiss>
eric_engestrom: I honestly don't know. I don't see failed jobs in sidekiq, so it's as if it hasn't happened. But better double check your MRs
<eric_engestrom>
I don't have any personally, but I'll check on other's that I saw had trouble
vx has quit [Quit: G-Line: User has been permanently banned from this network.]
vx has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
blatant has joined #freedesktop
jkhsjdhjs has quit [Quit: Error: Leaving not permitted]
jkhsjdhjs has joined #freedesktop
lsd|2 has joined #freedesktop
Leopold__ has quit [Remote host closed the connection]
Leopold has joined #freedesktop
ximion has joined #freedesktop
ximion has quit [Quit: Detached from the Matrix]
<daniels>
mripard: hmm, I'm wondering if it's better to have drm/{misc,intel,...}/kernel projects, or if we have drm/{misc,intel,...} projects containing the kernel, and like drm/acl/{misc,intel,...} groups for the user lists?
<daniels>
(this really is a bikeshed so feel free to ignore me)
<mripard>
I followed how xe was setup, but I don't have an opinion, really
<bentiss>
daniels: thatś an interesting one
<mripard>
I think I like the current setup because it allows to have other sub-repos for tools, or issues if we ever want to store our issues templates in a separate reop
<mripard>
but I have to admit I also didn't really grasp what you were suggesting to do instead :)
sergiomiguelrp has quit [Remote host closed the connection]
sergiomiguelrp has joined #freedesktop
Leopold__ has joined #freedesktop
jarthur has quit [Ping timeout: 480 seconds]
Leopold has quit [Ping timeout: 480 seconds]
scrumplex_ has joined #freedesktop
scrumplex__ has joined #freedesktop
<daniels>
bentiss: yeah, I couldn't make heads or tails of it, but the API request works, so shrug
<bentiss>
except that I'm not .schedule!, just .schedule
scrumplex has joined #freedesktop
<daniels>
yeah, .schedule returned false, so I used .schedule! which at least gave me the error
<bentiss>
but in the end the repo is moved, right?
scrumplex__ has quit [Ping timeout: 480 seconds]
mripard_ has joined #freedesktop
<daniels>
when using the API endpoint, yeah
<daniels>
never got it to work from rails
mripard has quit [Ping timeout: 480 seconds]
AbleBacon has joined #freedesktop
<daniels>
bentiss: current wtf is trying to do anything with repos throwing 'storage not found: "gitaly-2"', and inspecting Gitlab.config.repositories.storages[] only showing gitaly-0/default, yet https://gitlab.freedesktop.org/admin/gitaly_servers shows all 4 shards
blatant has quit [Quit: WeeChat 4.2.1]
<daniels>
(I'm pretty sure this is what's causing e.g. DELETE /api/v4/projects/2206/fork HTTP/2 from API to fail; ::Projects::UnlinkForkService.new(Project.find(2206), User.find(1)).execute from rails does the same and throws the 'storage not found' error)
<bentiss>
daniels: which pod are you running this?
<bentiss>
daniels: be sure to not run in one of the backups one, as they do not have visibility on all gitaly servers
<daniels>
oh ffs
<daniels>
yeah
<bentiss>
gitlab-prod-toolbox-7df5bdcbb-rxk47 should be good
<daniels>
yeah, I literally just found that as you said it - have updated my shell alias accordingly
<bentiss>
daniels: is it better now?
<daniels>
yeah, thankyou :)
<robclark>
daniels: did you know you cycled a bunch of drm/msm MRs from open to closed to reopen back to closed? Idk if that was just playing with API gone wrong, or??
<daniels>
hmm, the original close was inadvertent, the reopen was manually undoing those, and the next close was also inadvertent
<daniels>
sigh
<robclark>
heheh, no worries.. they seemed to have all happened quickly enough that it seemed automated
<daniels>
yeah, the closes were all automated and I wish they weren't :P wasn't supposed to happen
Juest has quit []
karolherbst has quit [Remote host closed the connection]
karolherbst has joined #freedesktop
cisco87 has quit [Quit: No Ping reply in 180 seconds.]
cisco87 has joined #freedesktop
zxrom has quit [Ping timeout: 480 seconds]
ximion has joined #freedesktop
pzanoni has joined #freedesktop
pzanoni has quit []
pzanoni has joined #freedesktop
tzimmermann has quit [Quit: Leaving]
Kayden has quit [Quit: -> JF]
sima has joined #freedesktop
zxrom has joined #freedesktop
mripard_ has quit [Remote host closed the connection]
ximion has quit [Quit: Detached from the Matrix]
nektro__ has joined #freedesktop
CutieMelo has joined #freedesktop
nektro_ has quit [Ping timeout: 480 seconds]
itaipu has quit [Read error: Connection reset by peer]