daniels changed the topic of #freedesktop to: GitLab is currently down for upgrade; will be a while before it's back || https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org
columbarius has joined #freedesktop
co1umbarius has quit [Ping timeout: 480 seconds]
Kayden has quit [Quit: home]
Kayden has joined #freedesktop
ANDROID_IOS_user has joined #freedesktop
todi has joined #freedesktop
ANDROID_IOS_user_ has joined #freedesktop
ANDROID_IOS_user has quit [Ping timeout: 480 seconds]
todi1 has quit [Ping timeout: 480 seconds]
ANDROID_IOS_user__ has joined #freedesktop
ANDROID_IOS_user_ has quit [Ping timeout: 480 seconds]
ximion has quit [Quit: Detached from the Matrix]
bmodem has joined #freedesktop
swatish2 has joined #freedesktop
AbleBacon has quit [Read error: Connection reset by peer]
ANDROID_IOS_user__ has quit []
ANDROID_IOS_user__ has joined #freedesktop
sima has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
bmodem has joined #freedesktop
bmodem has quit [Excess Flood]
bmodem has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
bmodem has joined #freedesktop
bmodem has quit [Excess Flood]
bmodem has joined #freedesktop
bmodem has quit [Ping timeout: 480 seconds]
bmodem has joined #freedesktop
An0num0us has joined #freedesktop
ANDROID_IOS_user__ has quit [Remote host closed the connection]
ANDROID_IOS_user__ has joined #freedesktop
<bentiss>
daniels: re queries: I haven't done time analysis for a while. Usually I just look for '"status": 500' or such, which is fast enough for me
<bentiss>
the cost of making it smarter by parsing fields and storing them in a db was too high for the litle gain IMO
tzimmermann has joined #freedesktop
tzimmermann has quit []
tzimmermann has joined #freedesktop
<bentiss>
runner-gating: so we are still seeing issues with it? I thought the switch back to debian solved most of the issues
<bentiss>
and given that the script already uses $CI_PRE_CLONE_SCRIPT, ${CI_JOB_TOKEN} I would say that we probably have access to all of the variables of the job when we execute
<bentiss>
regarding cache: I didn't want to cache on the runners so we could update the script without worrying to deploy it on the runner. A short term cache could be done though. But also I wanted to trust only the script on https://gitlab.fd.o so that we don't temper it in a job and then let the script kiddies come in.
<bentiss>
daniels: I'm all for improving that script. It is not meant to stay in stone
ANDROID_IOS_user__ has quit [Remote host closed the connection]
<daniels>
bentiss: ok, thanks :) makes sense, and yeah, indeed I can't find very many failures caused by the gating script at all atm
<eric_engestrom>
daniels: thanks for the ping; I'll check what I can see/do about it
ANDROID_IOS_user has joined #freedesktop
<daniels>
eric_engestrom: thanks!
<bentiss>
daniels: also I remembered *not* using CI_USER_* because you can override most variables in the job, so I don't trust it to be always set by gitlab
<eric_engestrom>
daniels: internet at that location is stable for everything else; I restarted the proxy, it didn't seem to help; I've checked that we don't have a disk full; I don't know what else to check, so I'm taking the farm offline until we can figure out what's wrong :(
<mupuf>
eric_engestrom: did you restart the host?
<mupuf>
You may also want to ask local admins to reboot the router
<mupuf>
And switches
<eric_engestrom>
I didn't; good idea, doing that now
<eric_engestrom>
the hsot reboot
pkira has joined #freedesktop
<eric_engestrom>
*host
<mupuf>
Check the load average on the server too
<mupuf>
Dmesg too
blatant has quit [Ping timeout: 480 seconds]
<eric_engestrom>
nothing in dmesg, load average is low (<1.5)
<mupuf>
Do you know if the issue is between the DUT and the proxy, or the proxy and the world?
<mupuf>
I assume even a cache hit would still require a HEAD on the URL
blatant has joined #freedesktop
<eric_engestrom>
between the host and the world
<eric_engestrom>
and it's not interrupted connections, it's _extremely_ slow
<eric_engestrom>
20-30 KB/s
<mupuf>
I had a case like this with the registry yesterday
<mupuf>
Took a minute to recover
<mupuf>
It was more like 8K every 10s
<mupuf>
Then it hit the turbo and downloaded the last 20MB in a second
<mupuf>
What proxy are you using?
<eric_engestrom>
nginx
<mupuf>
Maybe parallel downloads would help?
<eric_engestrom>
but it's not the proxy
<eric_engestrom>
it's between the host and internet
<eric_engestrom>
and not just fdo, any server
<mupuf>
Oh, so everything is slow, at all times?
<eric_engestrom>
I tried downloading the archlinux and fedora iso to test the bandwidth
<eric_engestrom>
yeah
<eric_engestrom>
I'm taking the farm offline
<mupuf>
Yeah, take it down
<mupuf>
Would be nice to know the bit error rate of the network interface
<mupuf>
I would propose changing the cable first, then the NIC
<eric_engestrom>
gitlab isn't letting me post a kernel bug backtrace? "Your merge request has been recognized as spam. Please, change the content to proceed."
<eric_engestrom>
do we have any control over gitlab's filter?
<bentiss>
eric_engestrom: not much. There is a spam checker pod, but it's rather gready and nobody knows if wwe can configure it
<eric_engestrom>
(also, this is while editing the description of an MR, and the error message is badly phrased and makes it sound like the commits are the issue)
<eric_engestrom>
bentiss: ack; not sure what to do then :]
<bentiss>
eric_engestrom: attach the file elsewhere (maybe try a snippet) and then put the link in the issue
<daniels>
please feel free to MR it - after my Fedora upgrade I can no longer run ci_run_n_monitor so the top commit isn’t going to land soon
daniels changed the topic of #freedesktop to: https://www.freedesktop.org infrastructure and online services || for questions about freedesktop.org projects, please see each project's contact || for discussions about specifications, please use https://gitlab.freedesktop.org/xdg or xdg@lists.freedesktop.org