ChanServ changed the topic of #asahi-dev to: Asahi Linux: porting Linux to Apple Silicon macs | Non-development talk: #asahi | General development | GitHub: https://alx.sh/g | Wiki: https://alx.sh/w | Logs: https://alx.sh/l/asahi-dev
<nicolas17> okay so, what if I want to run *Linux* under the m1n1 hypervisor and trace its hardware access?
wrender has quit [Ping timeout: 480 seconds]
<j`ey> nicolas17: run_guest.sh will help
fossdd_ has joined #asahi-dev
fossdd__ has joined #asahi-dev
fossdd___ has joined #asahi-dev
<nicolas17> I'm trying linux.py, with the kernel, dtb and initramfs copied from the Fedora installation
<nicolas17> after "TTY> Preparing to boot kernel at" the serial connection dies and the machine reboots
<nicolas17> I wonder if I have the wrong root dev
fossdd has quit [Ping timeout: 480 seconds]
<nicolas17> ohh, this guide was written for arch
<nicolas17> now I'm on fedora with btrfs, I need to give it the btrfs subvolume name
fossdd_ has quit [Ping timeout: 480 seconds]
fossdd__ has quit [Ping timeout: 480 seconds]
fossdd___ has quit [Ping timeout: 480 seconds]
<nicolas17> USB0: shutdown
<nicolas17> usb-dwc3@382280000: timeout while waiting for DWC3_DSTS_DEVCTRLHLT during shutdown
<nicolas17> USB1: shutdown
<nicolas17> is the last stuff I see on screen before it reboots
<nicolas17> (the reboot is so quick that the serial interface dies before it can send all that, I had to record a video of the screen)
fossdd has joined #asahi-dev
fossdd_ has joined #asahi-dev
fossdd has quit [Ping timeout: 480 seconds]
<nicolas17> mmmmaybe the problem is this kernel is an EFI binary and I need a "raw" kernel image? :<
al3xtjames has quit [Quit: al3xtjames]
KxCORP has quit [Quit: Bye!]
KxCORP has joined #asahi-dev
Bertrand___ has quit [Ping timeout: 480 seconds]
Bertrand___ has joined #asahi-dev
<nicolas17> oh well run guest wants a Linux build directory so I have to build a kernel from source *anyway*
<tpw_rules> it didn't when i last used it, like a couple years ago
<tpw_rules> you can feed it symbols and stuff which is nice
<nicolas17> "Use this utility script to properly boot a Linux kernel under the m1n1 hypervisor: tools/run_guest_kernel.sh"
<nicolas17> "Note that the script expects the path to the Linux build tree (i.e. the directory where .config and Makefile are). It will pick out the kernel and corresponding device trees from within."
<tpw_rules> i manually invoked that stuff
<nicolas17> ah I see, run_guest_kernel figures things out from the build tree and calls run_guest
<tpw_rules> yeah
<nicolas17> also it's chainloading a second m1n1 which I may not need
<nicolas17> but anyway is linux.py or run_guest.py supposed to work with an EFI image?
<tpw_rules> i wouldn't imagine so
<nicolas17> then I need to build my own because the stuff I found in fedora's /boot is an EFI image ^^'
<tpw_rules> do they only ship it that way? i wonder if there's a way to unpack it
<tpw_rules> kind of rude if so i would say, but maybe that makes sense for a *stares judgementally* binary distro
<tpw_rules> :P
<nicolas17> I made the mistake of starting to build it on my x86 laptop
<tpw_rules> not cross?
<nicolas17> yes, using a cross compiler
<nicolas17> but I bet gcc would go brrrr much faster on the M2
<tpw_rules> ah, then what's the mistake
<nicolas17> i5-8250U vs Apple M2
<nicolas17> wish I could know progress%
john-cabaj has quit [Ping timeout: 480 seconds]
<nicolas17> tpw_rules: while waiting for the build to finish I started it on the M2 too, it finished in 18 minutes and the x86 cross-build is still going; see, that's why it was a mistake :P
<nicolas17> oh wait it effing failed
<nicolas17> it was missing pahole to process debug info or something
nickchan has quit [Quit: ZNC 1.8.2+deb3.1 - https://znc.in]
nickchan has joined #asahi-dev
<nicolas17> hm, in a normal boot, m1n1 starts u-boot which starts grub which starts linux... here m1n1 directly starts linux, how does that work? I thought uboot and grub did extra initialization that was necessary for linux?
<tpw_rules> not really
<tpw_rules> certainly not grub, u-boot doesn't iirc in the apple silicon scenario
<nicolas17> I got the kernel to boot but it can't find the root filesystem
<nicolas17> it drops me into "press enter for maintenance shell" but the keyboard doesn't work either
<tpw_rules> does a USB one?
wrender has joined #asahi-dev
<nicolas17> my cheap crappy USB keyboard has a damaged cable (I get "unable to enumerate USB device" on this x86 laptop and this time I couldn't get it to work by wiggling the cable)
<nicolas17> maybe *fedora's kernel configuration* doesn't work in this scenario and I should use https://github.com/AsahiLinux/PKGBUILDs/blob/main/linux-asahi/config instead
<tpw_rules> are you missing a kernel command line argument? idk how fedora finds the rootfs
<nicolas17> my original kernel build on the i5 is still going btw, at this point I'm just curious how long it will take :D
<nicolas17> I tried "rootwait root=UUID=15636f9a-6784-410d-8635-e6c9f9d7adf5 rootflags=subvol=root" (which is almost what the normal boot has) and it says it can't find /dev/disk/by-uuid/156...
<nicolas17> so it's not a btrfs issue, it's not even finding the block dev
<tpw_rules> oh, so maybe there is an issue detecting the disk
<tpw_rules> i don't think u-boot has anything to do with that, but maybe it inits the nvme or something and there's a timeout?
<nicolas17> wonder if this kernel has nvme as a module... but wouldn't it get the module from the initramfs then?
<tpw_rules> assuming you built it into there and it's the same version and stuff
<nicolas17> ...yeahhh I probably have a big version mismatch
<nicolas17> I was just going to use fedora's kernel and initramfs from /boot
<nicolas17> but the kernel is EFI and I need a plain image
<nicolas17> so I built my own kernel from the asahi-linux repo... and tried booting it with fedora's initramfs
<tpw_rules> why not just get the same kernel source
<nicolas17> in hindsight that's probably not going to have matching versions >_<
<nicolas17> where can I find that? I don't know how to fedora :<
<tpw_rules> hold on i figured all that out and wrote it down for myself
<tpw_rules> idk if you can build it like a normal linux tree, i never tried
<tpw_rules> but copr i think is what builds the .rpms and the latest ver there matches the latest tag in that repo
<tpw_rules> so i'm 95% sure it's the source and what users are actually installing. would be nice to figure out how to query dnf and check
<tpw_rules> (never used dnf either i don't think)
* nicolas17 clones repo and meanwhile watches how his original cross-build is *still going*
jeisom has quit [Ping timeout: 480 seconds]
Bertrand___ has quit [Read error: Connection reset by peer]
Bertrand___ has joined #asahi-dev
<nicolas17> oh also I get no serial console, USB disconnects as soon as the kernel starts
<nicolas17> ah serial console is only with hv
<nicolas17> okay! I have a serial console via run_guest
<nicolas17> x86 crossbuild just finished, 165 min XD
tristan2 has joined #asahi-dev
tristan2_ has quit [Ping timeout: 480 seconds]
wrender has quit [Ping timeout: 480 seconds]
gladiac is now known as Guest2584
gladiac has joined #asahi-dev
Guest2584 has quit [Ping timeout: 480 seconds]
<nicolas17> I think I got it, Fedora's EXTRAVERSION and my own didn't match so it couldn't load any modules from the initrd
<nicolas17> okay this is too fragile... I need to figure out how to build my own initramfs from my own built modules
Bertrand___ has quit [Ping timeout: 480 seconds]
Bertrand___ has joined #asahi-dev
mf54 has quit [Remote host closed the connection]
mf54 has joined #asahi-dev
jeisom has joined #asahi-dev
WindowPain has joined #asahi-dev
WindowPa| has quit [Read error: Connection reset by peer]
jeisom has quit [Ping timeout: 480 seconds]
john-cabaj has joined #asahi-dev
<zzywysm> nicolas17: everyone looks at me weird when i try to slim down kernel configurations for Apple Silicon, but i suppose a 165 minute build must be thrilling for some
<j`ey> zzywysm: not at all, I also spent a long time working on a slim build :P
<zzywysm> in the future, consider trying this instead: https://github.com/AsahiLinux/docs/wiki/Reference-Asahi-kernel-config
<j`ey> I probably went further than you since I also did source changes!
<zzywysm> j`ey: if you've got a link to those source changes, i'd love to peek
<j`ey> nicolas17: also if by EFI image you mean the kernel file.. that should be fine to boot
<j`ey> the EFI header is also executable arm64 assembly!
<j`ey> zzywysm: no sorry just did it locally, I can maybe look at sending you some bits (I dont think it's worth actually doing, was just for fun)
<maz> j`ey: by luck!
<zzywysm> j`ey: what were the nature of the source changes?
<j`ey> ccmp x18, #0, #0xd, pl, does nothing, but encodes to MZ!
<j`ey> zzywysm: stuff like enabling the LSE instructions at compile time, removing a bunch of the perf tables that werent for M1
<j`ey> I tried to compile a kernel without nvhe KVM.. but that stuff is tangled quite firmly
<zzywysm> i like KVM, and use a M1 most of the time
<j`ey> zzywysm: M1 doesn't support the nvhe mode
<zzywysm> i'm afraid to disable containers because i suspect NixOS would crap a brick
<j`ey> (a particular mode of KVM, it supports the other mode)
<j`ey> nvhe = arm 8.0 virtualisation, vhe = 8.1+ virtualisation
<maz> I really don't want to have to ever split the two. specially when people use nVHE on 9.x HW.
<zzywysm> i'm of the philosophy that every kernel configuration option you can turn off is a return-oriented-programming gadget that no longer exists
<maz> zzywysm: sure. but the maintenance cost is way too high, and completely impractical.
<zzywysm> maz: the maintenance cost is basically "every time a new kernel release gets cut, you do a make oldconfig and answer a couple questions"
<maz> no: the maintenance cost is that an every *patch* you apply, you test *all* the possible combinations.
<maz> that's what maintenance means.
kraem has quit [Remote host closed the connection]
<zzywysm> maz: but by drastically reducing the size of the kernel configuration, you also drastically reduce the size of the exponential tree of testing you need to be doing
<zzywysm> code that isn't built can't hurt you
<maz> zzywysm: size doesn't matter in that case. I can compile a kernel in a couple of minutes. testing takes hours, multiplied by the number of architectural variants this needs to run on.
<j`ey> maz: yes I was just trying to split it for fun, but it was a case of never-ending errors, so I gave up
<j`ey> zzywysm: I agree if you have some fixed hw for whatever reason, but of course general purpose distros cant do that
<maz> j`ey: a better option is to keep building it, but to reclaim the section at boot time. I had patches for that, but it was something like a couple of pages, which wasn't worth the complexity.
kraem has joined #asahi-dev
<j`ey> makes sense
kraem has quit [Remote host closed the connection]
<zzywysm> the next step for my personal Apple Silicon machines is getting the Asahi firmware into CONFIG_EXTRA_FIRMWARE so i can go module-less
<zzywysm> mmmm, delicious module-less-ness
<j`ey> zzywysm: ah yeah, I did that too
chadmed has quit [Quit: Konversation terminated!]
chadmed has joined #asahi-dev
chadmed has quit []
chadmed has joined #asahi-dev
john-cabaj has quit [Remote host closed the connection]
john-cabaj has joined #asahi-dev
chadmed has quit []
chadmed has joined #asahi-dev
chadmed has quit []
chadmed has joined #asahi-dev
chadmed has quit []
kraem has joined #asahi-dev
chadmed has joined #asahi-dev
chadmed has quit []
chadmed has joined #asahi-dev
<zzywysm> chadmed is likely asleep and doesn't realize a gremlin is repeatedly plugging and unplugging his Ethernet cable
chadmed has quit []
chadmed has joined #asahi-dev
chadmed has quit []
chadmed has joined #asahi-dev
dylanchapell has quit [Read error: Connection reset by peer]
<buroa> How can we build m1n1 using musl instead of glibc?
<sven> what makes you think m1n1 needs glibc?
<buroa> That's what is used to compile it.
<buroa> I want to compile it using musl, so I can build it into Talos Linux's toolchain, which uses alpine.
<j`ey> buroa: it doesnt use libc
dylanchapell has joined #asahi-dev
<buroa> It doesn't use musl, so what does it use?
<j`ey> nothing
<buroa> Lol
<buroa> Ok
<buroa> libc.
<buroa> I don't have libc in my build as I use musl. By default that is using libc, so it's building.
<j`ey> not sure what the issue is, but alpine has m1n1, that uses musl, so it works
<maz> dragging libc in something as low-level as m1n1 would be... interesting.
<buroa> So building it with musl gets all kinds of errors... error: implicit declaration of function 'atol' [-Werror=implicit-function-declaration]
<buroa> There are many more, but that's just the surface.
<j`ey> that's defined in src/string.{c,h}
<sven> sysinc/string.h i think
<j`ey> sorry yeah, src/string.c sysinc/string.h
<j`ey> buroa: as I showed, alpine builds it just fine, so something weird in your setup
<buroa> It's... very custom
<buroa> Once I get this all figured out, I'll be able to publish Talos Linux versions for the community.
<janneg> can you paste the full failing compile command? i.e. the last compile cmd line from `make V=1`
<janneg> can you grep in /toolchain/lib/gcc/aarch64-linux-musl/13.2.0/include for "STRING_H"
<buroa> Yep, sec
<j`ey> nice spot janneg
<janneg> that file shouldn't be in the compiler include path "/toolchain/lib/gcc/aarch64-linux-musl/13.2.0/include"
<janneg> buroa: please paste build/display.S after running https://privatebin.net/?0d1d6390d12767fc=#6iGTh2LLnGYGDBkZ2yXtVvQ52RHeKPP11LnSE9PNmJJD
<zzywysm> when people tell me that the musl movement isn't a compulsion, i will remember this :D
<janneg> I don't think this is related to musl
<zzywysm> janneg: he says "So building it with musl gets all kinds of errors..." not realizing that neither musl nor glibc are necessary to build m1n1
<sven> let's focus on the problem instead of musl vs. glibc
<zzywysm> agreed
<zzywysm> he's building with -ffreestanding and -Werror=implicit-function-declaration simultaneously, are those mutually compatible?
<sven> yeah, and that's also just from the normal m1n1 Makefile
* janneg has built m1n1 on macos with https://github.com/richfelker/musl-cross-make
<sam_> yes, they are mutually compatible
<sam_> you need prototypes for things either way
<sam_> it's just about where they come from :)
<janneg> why would -ffreestanding be incompatible with -Werror=implicit-function-declaration? the latter just errors out if an used function misses an declaration
<sam_> exactly
<sam_> (and it's default in gcc 14+, clang 16+)
<janneg> m1n1 has prototypes for every used libc function in sysinc/
<zzywysm> ah, cool
jeisom has joined #asahi-dev
Bertrand___ has quit [Read error: Connection reset by peer]
Bertrand___ has joined #asahi-dev
<sven> weird, it includes /usr/include/string.h for some reason
<sven> what does /toolchain/bin/gcc --version print?
<janneg> that would mean that "-nostdinc" is somehow broken
<sven> yeah, i'm curious which compiler that is
<janneg> fedora's 13.2.1 works as expected
<janneg> try adding "-nostdinc++", that was added in gcc-13 but it would be strange if that borked gcc -nostdinc
<janneg> please add "-v" to the command to generate display.S and paste the output from the command line
<sven> huh, where does -isystem /usr/include come from?
<sven> oh, wait, you can that with just -nostdinc++
<sven> please run that -v thing again with -nostdinc present
<buroa> $(shell $(CC) -print-file-name=include) I think: https://github.com/AsahiLinux/m1n1/blob/main/Makefile#L50C21-L50C60
<sven> no
<sven> we would've seen that earlier when you can make V=1
<sven> *ran
<janneg> no, that outputs /toolchain/lib/gcc/aarch64-linux-musl/13.2.0/include
<janneg> please add -nostdinc++ instead of replacing -nostdinc
<janneg> but I would expect the output to be the same
<sven> are you sure? i just see -nostdinc++ there and no -nostdinc
<buroa> Wait, you want both?
<janneg> yes
<buroa> Sec
<j`ey> buroa: I would look at your gcc build see what flags youre passing..
<sven> this smells like a messed up toolchain
<sven> can you paste /toolchain/lib/gcc/aarch64-linux-musl/13.2.0/specs ?
<janneg> there is nothing suspicious in there. config is printed first
<janneg> or the output of `/toolchain/bin/gcc -dumpspecs`
<sven> you can probably add that -isystem /usr/include by messing with that file
<sven> and if this toolchain was never used to build with -nostdinc no one would've noticed that bug
<j`ey> I think the kernel uses nostdinc..
<sven> so much for that theory :D
<sven> that looks very suspicious
<sven> that seems to add -isystem /include for the c pre-processor
<sven> hrm, that's not /usr/include though
<janneg> `ln -s /toolchain/include /usr/include` looks a little suspicios as well
<janneg> with that adjustment the "--with-native-system-header-dir=/toolchain/include" from the gcc configure line looks suspicious as well
<sven> this very much sounds like a broken toolchain to me
<sam_> how did this thing get built
<janneg> yes, to me as well. only open question is how did break
<janneg> +it
<sam_> oh boy it's that script? adjust.sh?
<sven> duct tape and lots of luck? :D
<sam_> not enough!
<sven> do you know about that script? :D
<sam_> all i know is what i just read from some of the above links and i feel a bit ill
<sven> though i guess that repo is archived so maybe it's not used anymore?
<j`ey> anyway, not a m1n1 issue :P
<sven> yeah
<zzywysm> why can't i link musl into these arm64 assembly files?
<janneg> current toolchain repo seems to be https://github.com/siderolabs/toolchain
<janneg> and I don't see any gross hacks uin there
wrender has joined #asahi-dev
<sven> at this point it's either something else inside that spec file or a gcc that was somehow patched i think
wrender has quit [Ping timeout: 480 seconds]
<buroa> Ok, I got it to build.
<buroa> Last step: I need to concat m1n1 and u-boot.
<janneg> buroa: how?
<janneg> don't forget the dtbs
<buroa> That's what I wanted to ask you guys.
<nicolas17> no, how did you get it to build :D
<buroa> And I stopped using the base and just depended on the tools. It's complicated
<buroa> I built it exactly how Talos builds the kernel stage, essentially
<buroa> It must do something different with the toolchain.
<buroa> I see that we have some Apple stuff inside u-boot upstream: https://docs.u-boot.org/en/latest/board/apple/m1.html#image-installation
<buroa> This is how Talos builds a lot of their u-boot things.
<j`ey> youl probably want to use the downstream uboot fork
<buroa> You can probably follow along here: https://github.com/siderolabs/pkgs/blob/main/u-boot/pkg.yaml
<buroa> You guys seem smart, lol, so it should make sense :P
<buroa> Anyways, I'm at the last step, and I have m1n1.bin from previously built stage... I want to include all of the .dtb's somehow (is that possible) for all the systems we support
<buroa> Something like this: cat /m1n1/m1n1.bin ${APPLE_M1_U_BOOT}/u-boot-nodtb.bin?? > /rootfs/apple_m1/u-boot.bin
<j`ey> yup
<j`ey> we have a script for it!
<buroa> Talos does things extra, so I have to craft this a-like. Let me take a look at those.
<buroa> They do include a dbts folder... do I put them in there?
<zzywysm> dtbs get cat-ted in, in between m1n1 and u-boot
<j`ey> no, the dtbs go in the m1n1+uboot image
<buroa> ./usr/install/arm64/dtb/apple
<buroa> There are multiple dtbs though, how do I make one unified u-boot.bin with them?
<j`ey> buroa: see the update-m1n1 script
<buroa> Ahh! Just cat them all together!
<buroa> Siiiimple
<janneg> remember to compress u-boot with gzip. that is mandatory since m1n1 can't determine size of the u-boot binary
anders_2 has joined #asahi-dev
jeisom has quit [Ping timeout: 480 seconds]
<buroa> Thanks for everyone's help here!
<buroa> I'll let you know if I get this booted up :-)
<buroa> Those were my two blockers. So really, thanks again
<j`ey> good luck!
john-cabaj has quit [Quit: john-cabaj]
anders__2 has quit [Ping timeout: 480 seconds]
john-cabaj has joined #asahi-dev
zzywysm has quit [Quit: Textual IRC Client: www.textualapp.com]
zzywysm has joined #asahi-dev
___nick___ has joined #asahi-dev
___nick___ has quit []
___nick___ has joined #asahi-dev
jeisom has joined #asahi-dev
__gilles has joined #asahi-dev
WindowPa- has joined #asahi-dev
WindowPain has quit [Ping timeout: 480 seconds]
jeisom has quit [Quit: Leaving]
jeisom has joined #asahi-dev
___nick___ has quit [Ping timeout: 480 seconds]
jeisom_ has joined #asahi-dev
jeisom has quit [Read error: Connection reset by peer]
wrender has joined #asahi-dev
jeisom_ has quit [Ping timeout: 480 seconds]
jeisom has joined #asahi-dev
wrender has quit [Ping timeout: 480 seconds]
wrender has joined #asahi-dev
wrender has quit [Ping timeout: 480 seconds]
wrender has joined #asahi-dev
wrender has quit [Ping timeout: 480 seconds]
commandoline has joined #asahi-dev
commandoline_ has quit [Read error: Connection reset by peer]
wrender has joined #asahi-dev
wrender has quit [Ping timeout: 480 seconds]
wrender has joined #asahi-dev
<nicolas17> j`ey: I tried copying /boot/vmlinuz-6.6.3-413.asahi.fc39.aarch64+16k from Fedora Asahi to my other computer and booting that with run_guest_kernel.sh, it throws "ERROR: Devicetree found but no kernel."
<nicolas17> gzipping it doesn't help
<nicolas17> PE32+ executable (EFI application) Aarch64 (stripped to external PDB), for MS Windows
<nicolas17> ok you know what does work
<j`ey> yeah, I had run similar files with m1n1
<nicolas17> except with kernel modules disabled
<nicolas17> so everything is built in and the incompatible modules in the initramfs are ignored
<nicolas17> hm I have keyboard but not trackpad
<nicolas17> and no touchbar or wireless... must be firmware then
<nicolas17> "You can ignore the Arch Linux initramfs, but you should instead provide an initramfs with /lib/firmware/{brcm/apple} to make sure the built-in modules can find their firmware early"