<ynezz>
f00b4r0: there was similar report on #openwrt week ago, I was unable to reproduce it, resolved after a hour or two
<ynezz>
f00b4r0: feel free to create a ticket with your findings, attaching the broken files, so anyone interested could see where the desync happened etc.
<ynezz>
f00b4r0: IIRC interestingly enough it was for the same arch
<ynezz>
f00b4r0: for host in downloads mirror-03.infra; do curl -s https://$host.openwrt.org/releases/23.05.3/packages/mipsel_24kc/packages/Packages | tee $host.packages | sha256sum; done
<ynezz>
yields same e086a0c47ffa89f15535eaefc90afc97ba0ad2786f60a5808a91669379d02cf7 here
<ynezz>
I just tried it from a two different datacenters in .fr and unable to reproduce
<ynezz>
X-Served-By: cache-fra-etou8220116-FRA, cache-ams2100132-AMS and X-Served-By: cache-fra-etou8220116-FRA, cache-lcy-eglc8600064-LCY
<Habbie>
f00b4r0, i'd be very curious to hear where your failing data is coming from
<Habbie>
(in the next window in my client, people are debugging bitflips in an Amsterdam data center)
<Habbie>
(might be entirely unrelated of course)
<ynezz>
from the DO server in AMS I'm having the same "x-served-by: cache-fra-etou8220118-FRA"
<ynezz>
ah, its different 116-FRA vs 118-FRA
<jow>
maybe some cahce purge didn't went through
<ynezz>
from helsinki x-served-by: cache-fra-etou8220116-FRA, cache-hel1410030-HEL, London: X-Served-By: cache-fra-etou8220116-FRA, cache-lcy-eglc8600078-LCY
<ynezz>
so seems to be similar POPs
<ynezz>
serving a valid content
<ynezz>
its quite interesting, that they server most of the requests from france, even from .cz its cache-fra-etou8220095-FRA, cache-fra-eddf8230100-FRA
<dwfreed>
FRA is not france, it's Frankfort, Germany
<dwfreed>
s/fort/furt/
<ynezz>
ah, ok
<dwfreed>
FRA is the IATA code for the Frankfurt airport; EDDF is the ICAO code; ETOU is the ICAO code for an army airfield just outside of Frankfurt itself; this is used to differentiate which specific DC in regions that have several
<dwfreed>
LCY is the IATA and EGLC is the ICAO code for London City Airport
<dwfreed>
It's very common to use IATA and ICAO codes as location references for datacenters; they're generally pretty well-known
<dwfreed>
unlike UN LOC codes
<Habbie>
and germany is not a weird place to serve europe from at all
<dwfreed>
Yeah, Frankfurt is an incredibly common location, given its relatively central position geographically
<ynezz>
yep, it makes sense now, thanks for enlightening me
hanetzer has joined #openwrt-devel
hanetzer1 has quit [Ping timeout: 480 seconds]
robimarko has joined #openwrt-devel
<f00b4r0>
ynezz: it resolved after about half a day. It affected only the packages feed signature
<f00b4r0>
ynezz: could we have a buildbot feeding an invalid sig every now and then?
<f00b4r0>
Habbie: I have unfortunately not track of the offending IPs but it happened over both v4 and v6
<jow>
f00b4r0: do you have a copy of the affected file?
<f00b4r0>
jow: no because when the sig fails the system deletes the file anyway
<jow>
ok. I believe a cache coherency issue is more likely than wrong generation on the buildbot side
<jow>
Packages.sig and Packages.gz got likely cached at different times in the middle of an upload
<jow>
afair there is supposed to be some cache purging call in the rsync post xfer hook, but maybe it is not reliable
<jow>
or maybe it needs to be repeated later to give fastly's infrastructure time to propagate the file updates internally
<SnarlingFox>
Morning all
<jow>
not sure if all their caches pull from the origin or if there's some internal cache hierachy
<jow>
conceptually this remains an unsolvable problem (maybe you remember our previous discussion wrt. to in-place rpo updates)
<jow>
the client would need to become smarter and index files should gain some form of version indicator
<jow>
e.g. Packages.$YYYYMMDDSSSS.sig
<Habbie>
if you do $YYYYMM.. now, plus symlinks, at least the problem becomes easier to debug
<Habbie>
(and then the client can become smarter later, optionally)
<jow>
yeah
<jow>
maybe also add a reference to the specific Packages.$YYYY... version in the Packages.sig freetext comment
<jow>
then make the client download Packages.sig first, discover the related Packages.XXX from it, and proceed to fetch that
<jow>
if client happens to fetch a stale Packages.sig, it will fetch the related old Packages.XXX version, if it happens to fetch the current Packages.sig, it'll fetch the current Packages.XXX that goes along with it
<jow>
but it'll likely only defer the problem to the later .ipk download stage
<jow>
when a stale Packages happen to reference .ipk which are not there anymore (although it would be less likely as not all .ipks vanish every time, and there's a chance that a cache serving stale Packages also servers stale .ipk files still)
<SnarlingFox>
robimarko: Pretty satisfied latest rebuild of WSQ50 firmware with ct-full-htt is stable, 48 hours now of CCTV streaming and still going strong. Just making one minor tweak to the LED currents and will then do another force-push for review.