fakuivan has quit [Remote host closed the connection]
fakuivan has joined #openwrt-devel
slh has quit [Quit: leaving]
slh64 has quit [Quit: gone]
slh has joined #openwrt-devel
<norris>
so, are github packages.git builders somehow using the host distro tools (like tar)? it took me a while, but I figured out they're not using this patch:
<ynezz>
norris: and since we've switched most of the build workers to Debian 11 recently it might explain this PKG_MIRROR_HASH woes we've been seeing recently
<KanjiMonster>
ynezz: mh, I guess the sdk shouldn't even ship with a tar symlink (or any other (maybe) host/ditro used binary)
<KanjiMonster>
I also see a few host path leaks in the symlinks (e.g. xxd -> /builder/shared-workdir/build/scripts/xxdi.pl)
<ynezz>
well, if we look into original SDK content, there is our patched tools/tar properly bundled
<ynezz>
but prereq-build then `Checking 'tar'... updated.` overwrites that with host tar
<ynezz>
so probably f75204036c & co. introduced those regressions?
<KanjiMonster>
might be, but updating the links is also (potentially) necessary. the sdk I downloaded symlinks python3.9, but on my host I don't have python3.9, only python3.10
<KanjiMonster>
though the core issue is that the host tar produces different binaries, and not updating the symlink only hid the issue
robimarko has quit [Remote host closed the connection]
<ynezz>
I assume, that we ship multiple tool binaries in the SDK for the reason, so we should prefer those and do not update it with host alternatives
<KanjiMonster>
if using host binaries is fine, then replacing them with symlinks to host binaries shouldn't be an issue. If it isn't, then we shouldn't even check for host binaries in the !sdk case
goliath has joined #openwrt-devel
<KanjiMonster>
just because the host system building the sdk had a viable host tool does not mean the host system running the sdk has and vice versa
<ynezz>
well, in case of `tar` specifically you'll end up with chicken-egg problem
<KanjiMonster>
I don't quite follow
<ynezz>
you need tar to unpack source of tar
<ynezz>
IMO in SDK case, we shouldn't update symlinks if we ship the host tool binary staging_dir/host/bin/.$(TOOL).bin
<KanjiMonster>
mh, seeing that the sdk isn't capable of compiling host binaries itself, we probably should be shipping binaries for all tools we are prepared to compile ourselves, and not a selection based on the sdk build host
<ynezz>
that was the case, but its currently broken due to those recent prereq-build.mk updates
GNUmoon has quit [Remote host closed the connection]
GNUmoon has joined #openwrt-devel
<ynezz>
so in my case, the prereq run dropped bzip2, find, patch, tar and xargs tools from the SDK and started using the binaries from the host instead
<KanjiMonster>
AFAICT we ship the selection of binaries based on the host. If the host had a tar prereq prefers, then the SDK would have only contained a symlink, and not the binary
<KanjiMonster>
... or do we a weird thing were we first use the host tar symlinked, and then tools/tar replaces that with the binary?
<ynezz>
yes
<ynezz>
you need to do that, otherwise you wouldn't be able to unpack the sources
<KanjiMonster>
then the prereq check maybe just shouldn't replace binaries, and only update actual symlinks
<KanjiMonster>
-f should be false for symlinks, so keeping it shoudn't prevent symlinks from being updated
<ynezz>
probably we should enable it only for SDK
<ynezz>
we want to update the symlinks when running in buildroot, that was the point of the changes
<KanjiMonster>
dunno, for cases where we link to versioned binaries, updating symlinks might be desired in a rolling release
<KanjiMonster>
ynezz: so readding that -f check at least left the tar binary alone, but successfully updated other symlinks (python3.9 => python3.10)
<ynezz>
KanjiMonster: yep, seeing that as well
<norris>
dwfreed: Fwiw, openwrt is using the correct tar flags for reproducibility. (--numeric-owner is the interesting one here.) The problem is that the openwrt patch makes that arg behave differently than the Debian copy
<norris>
And yes, I'd assume more people are tripping on this reproducibility problem
<KanjiMonster>
norris: yeah, I first I misunderstood that as optionally using host-tar, but seeing that it eventually replaces that symlink with the self-built one (and the staging_dir/host/bin/tar is expected to be the self-built one), the issue is "only" that the binary gets replaced with a symlink
<KanjiMonster>
we could have an elaborate check whether the host tar is able to produce reproducable tars, but then we would need to teach the sdk to build tar if it doesn't
<colo>
maybe use python's tar module to generate the archives instead?
<colo>
(on all OSes/platforms, I mean. I guess that would be more consistent than different `tar` implementations)
dangole_ has joined #openwrt-devel
floof58 has quit [Ping timeout: 480 seconds]
floof58 has joined #openwrt-devel
<KanjiMonster>
colo: the issue is that potentially some host information leaks into the generated tar ball, which is the reason for the self-built / controlled tar that properly filters out these things
<KanjiMonster>
so quite likely it would be even less consistent
minimal has joined #openwrt-devel
rua has joined #openwrt-devel
fakuivan has quit [Remote host closed the connection]
Danct12 has quit [Quit: WeeChat 4.0.2]
dangole_ has quit [Quit: Leaving]
dangole has joined #openwrt-devel
robimarko has joined #openwrt-devel
schwicht has joined #openwrt-devel
rua has quit [Quit: Leaving.]
schwicht has quit [Ping timeout: 480 seconds]
rua has joined #openwrt-devel
schwicht has joined #openwrt-devel
schwicht has quit [Ping timeout: 480 seconds]
schwicht has joined #openwrt-devel
schwicht has quit [Ping timeout: 480 seconds]
Acinonyx has joined #openwrt-devel
schwicht has joined #openwrt-devel
schwicht has quit [Ping timeout: 480 seconds]
schwicht has joined #openwrt-devel
aiyion has quit [Ping timeout: 480 seconds]
schwicht has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]