marcan changed the topic of #asahi to: Asahi Linux: porting Linux to Apple Silicon macs | General project discussion | GitHub: https://alx.sh/g | Wiki: https://alx.sh/w | Topics: #asahi-dev #asahi-re #asahi-gpu #asahi-offtopic | Keep things on topic | Logs: https://alx.sh/l/asahi
jevinskie[m] has joined #asahi
al3xtjames has quit [Read error: Connection reset by peer]
al3xtjames has joined #asahi
bgb_ has joined #asahi
bps has quit [charon.oftc.net liquid.oftc.net]
apetresc has quit [charon.oftc.net liquid.oftc.net]
ave has quit [charon.oftc.net liquid.oftc.net]
skipwich has quit [charon.oftc.net liquid.oftc.net]
<Glanzmann>
kettenis: How is the status of OpenBSD on M1? Have you ironed out the SMP bug?
<kettenis>
no, SMP is still subtly broken, and there are some issues with NVMe as well still
pitust[m] has joined #asahi
<kettenis>
some more OpenBSD developers are getting involved, so I have been working on the installer for folks to be able to install on NVMe without bricking their machines ;)
<Glanzmann>
kettenis: I see, what about the other drivers? Is usb working, what about ethernet?
bisko has joined #asahi
<kettenis>
all works although the ethernet gets the default address programmed in the controller which I suspect is the same for all M1 machines
<kettenis>
some incompatible device tree binding changes coming up though, which is why I'm not advertising M1 support in OpenBSD much yet
bisko has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
apetresc has joined #asahi
bisko has joined #asahi
GreatGodvin has joined #asahi
<kettenis>
I am curious though if the SMP issues will also show up on Linux when booted via m1n1
<kettenis>
don't think anyone has used Linux in anger yet that way
<pipcet[m]>
doing what?
<kettenis>
doing parallel builds of things natively
<_jannau_>
kettenis: I haven't seen any during kernel compiles. booting linux directly from m1n1 with sven's dart/usb-c patches
<pipcet[m]>
i've done that a lot with my kernel (based on corellium)
<pipcet[m]>
but I'm not sure what I should be looking for
<kettenis>
I'm seeing core dumps from (mostly) clang and ld.lld
<kettenis>
was wondering if there are some chicken bits that m1n1 doesn't set
<_jannau_>
pipcet[m]: you're also not booting via m1n1
<pipcet[m]>
sometimes I am.
<kettenis>
but if an upstream kernel with sven's patches works fine, this must be an OpenBSD problem
<_jannau_>
my the kernel builds should have used gcc/binutils
EER has joined #asahi
<j_ey>
kettenis: what's the symptoms?
<kettenis>
core dumps typically early on when running C++ constructors before main()
<kettenis>
not yet pinned down whether it is TLB, cache or memory ordering related
<kettenis>
reviewed whether the kernel adheres to the "break-before-make" rules
<kettenis>
and inserted loads of strong memory barriers in the kernel that didn't seem to make a significant difference
choozy has joined #asahi
GreatGodvin has quit [Quit: GreatGodvin]
<marcan>
We do know that OSX configures some "interesting" nonstandard memory behavior, but I don"t *think* our chicken bits do. Also, that would probably break VMs unless they reset them in Hypervisor.framework
<sven>
arnd: i found the bit in the DART mmio space that tells us if they support bypass mode
<sven>
the pcie ones don't, the usb ones do
<sven>
so for the pcie ones i have a dma-ranges on the pcie bridge that limits DMA to the first 4G
<sven>
and then i just program a static linear map and tell the iommu api that this iommu really needs a bypass domain
<arnd>
I see. Have you checked all the other DART instances to see what those support?
<sven>
can't access the thunderbolt ones but i suspect they won't allow bypass either
<sven>
but all others seem to support bypass mode
<marcan>
I still think we outright shouldn't support putting those in bypass mode, or at most do it only with a nondefault, scary parameter
<arnd>
ok. I suppose bypass mode on the thunderbolt wouldn't make much of a difference in practice, since that is too dangerous for practical use
<marcan>
the tbt ones in particular
<arnd>
for the on-board devices, I don't see much of a problem
<marcan>
well...
<marcan>
you are extending the system attack surface to include Broadcom firmware
<marcan>
Broadcom *wifi* firmware
<marcan>
that's a *significant* downgrade
<marcan>
so if we do that at least I want a scary kmsg warning
<arnd>
ah, indeed
<arnd>
I was only thinking of ethernet and pci-xhci
<marcan>
yeah, if we find a sane way to disable wifi in this scenario I'd be happier
<arnd>
though both of those also run firmware that might be attacked in theory
<marcan>
though really nobody should be using these things with 4K pages, given all this, except for like installers
<marcan>
yeah
<marcan>
but the wifi attack surface is a couple orders of magnitude worse (and it's had a bunch of CVEs already)
<sven>
how many distro ship 16k pagesize kernels these days?
<marcan>
we're going to change that ;)
<arnd>
zero
<marcan>
enough people will want M1 not to suck
<marcan>
and besides, 16K has advantages on other platforms too
<marcan>
do we even know if the GPU MMU does 4K pages? because if it doesn't, lol no 3D for 4K kernels
<arnd>
Most developer boards still have only Cortex-A53 or Cortex-A72, which don't support 16KB pages at all
<marcan>
I expect distros to ship multiple kernels
<marcan>
some already do
<marcan>
e.g. ubuntu has 64k and 4k options
<arnd>
I expect distros to fight that very hard, as it adds a lot of extra validation work to have more than one
<sven>
all we know about the gpu mmu is that it's not a dart
<arnd>
I honestly don't know why Ubuntu did it
<marcan>
*shrug*
<marcan>
I mean, look, if there is no better solution, that's that
<marcan>
if we can't make the kernel somehow deal with a 16K IOMMU when built for 4k, there is no other good option
<marcan>
and then if the distros don't ship 16k kernels, someone else will
<marcan>
people will use them
<sven>
i'd be happy with 4k without thunderbolt as a fallback. that way most people can at least use this
<marcan>
and then it's up to the distros whether to eventually cave or not
<sven>
and i'm sure there will be third party repos that will ship the distro kernels with 16k support then
<marcan>
sven: I'd also kill wifi for "encouragement" ;)
<arnd>
I still hope that opening up the discussion again will at least make some users understand just how bad 64K pages are for general-purpose workloads
<marcan>
insmod dart.ko expose_me_to_broadcom_firmware_bugs=1 (or whatever module we put the gate in)
<arnd>
though a more likely outcome is that a lot of users run microbenchmarks that show 64K pages being faster because of lower TLB pressure
<sven>
it might be possible to make the kernel deal with 16k iommu when built for 4k but the dart driver itself is already >1k loc or so
<marcan>
sounds like it's more of a kernel issue
<sven>
at least it boots this way. we can always tackle the 16k support later
<marcan>
allocating DMA memory with higher than page size alignment
<marcan>
I just want the wifi thing to be opt-in
<marcan>
people should be *aware* that running in this mode is crap for security for several reasons, but especially that one
<marcan>
if distros want to flip the flag instead of ship 16k, that has to be an explicit decision
<arnd>
yes, makes sense (on wifi and thunderbolt)
<arnd>
sven: right, that fixing the common iommu code to work with this should be possible by the time that distros actually want to ship installers to work on mac. That is still a going to be a while, and your solution is enough for an initial merge
<arnd>
of the iommu driver
<marcan>
as long as it's gated behind a nondefault argument I'm fine with that
<marcan>
we can just have a dart.enable_bypass=1 or whatever for now
<marcan>
and say you need that for usb/eth on 4k kernels
<marcan>
and then we can be more specific about just breaking wifi by default if we want to in the future
<j_ey>
sven: not too bad, arm-smmu-v3.c is over 3000 for example
<arnd>
for the streaming mappings, I think there is already a small security hole regardless of the iommu type and the page size, as dma_map_single() often gets passed small buffers that live in the same page as other (possibly sensitive) data
<sven>
j_ey: the point is that reviewing 1k lines of code is already quite a lot. adding the 16k/4k thing on top that makes it even worse
<arnd>
especially on USB, the kernel often just passes buffers that are only a few bytes long
<j_ey>
sven: oh right, I was thinking as a later patchset
<sven>
j_ey: oh, sure.
<sven>
dart also supports subpage protection fwiw, but patching *that* into the iommu code would be more annoying
<arnd>
drivers using only the coherent interfaces (dma_alloc_coherent() etc), currently get full pages, so those are safe as long as the page size matches, or we change the common code to use io page size allocations
<arnd>
sven: what are the parameters for sub-page protections? Can you e.g. map an arbitrary naturally aligned 64 byte range in a page?
<arnd>
(that would be the minimum the kernel might pass)
<sven>
i think so. i haven't tested this in detail but from what i can tell you can limit access inside the page down to that level
<arnd>
ok, perfect. So the dart would actually end up being more secure than most others for the streaming API ;-)
<sven>
:D
<arnd>
do if we find a way change the iommu ->map() callback to pass the location within the page and do sub-page protection as well as fix the dma-iommu code to no longer assume iopagesize<=page_suze, I think that should allow safely using wireless and thunderbolt with 4KB pages, it would only prevent you from using device assignment into VM guests and user space like DPDK
<sven>
agreed
<sven>
so i don't think the iommu_pagesize > cpu_pagesize fix is going to require a lot of changes, but it'll certainly require reviewing the entire code to be comfortable it doesn't break anything by accident
<sven>
(it kinda works by accident already if you just remove the BUG_ON)
<arnd>
sven: have you found out if dart can do both cache-coherent dma mappings and non-coherent ones? Is bypass mode still coherent?
<marcan>
arnd: by the way, "small buffers that live in the same page as other (possibly sensitive) data" isn't a "small" security hole
<marcan>
this is how the ps4 got owned ;)
<marcan>
so yes, let's look into subpage protection
<marcan>
doesn't need to happen for the initial submission of course
<marcan>
but this is very much worth doing
<brentr123[m]>
I wonder if the ps5 can get pwned the same way ;)
<arnd>
marcan: I wonder if it makes sense to address this as a general security issue then, and start by making the dma-iommu code use bounce buffers as an option, and make the sub-page protection as a way to make it faster
<bloom>
ooi what's the issue with 64k pages?
<bloom>
for general purpose workloads
<bloom>
arnd: ^
<arnd>
bloom: mostly a waste of memory, e.g. when dealing with small files your page cache often blows up to 5x the memory usage
<sven>
arnd: from what i can tell it always maps everything as coherent.
<sven>
bypass mode hasn't broken anything yet while using usb but i'm not confident there yet
<arnd>
sven: ok, good
<bloom>
arnd: Ah, got it.
<sven>
arnd: afaik the iommu code already has the option to use bounce buffers for untrusted pci devices
<marcan>
arnd: certainly, if we can make it use bounce buffers easily for sub-page maps, it makes sense to start off with that
<sven>
or maybe that was just a patchset i saw, let me check
* marcan
off to sleep
<arnd>
bloom: it's extremely workload specific, and a number of other things also go wrong: anything that small bits of data but requires page alignment can explode. Other things I'm sure cause problems are swapspace handling, and network buffers
<arnd>
bloom: ok the other hand, anything that is purely CPU bound and fits within a small working set tends to benefit from the lower TLB usage, so microbenchmarks almost always show 64KB pages come out winning
<arnd>
When I tested kernel builds on Amazon EC2 comparing page sizes, I found that with 4GB RAM per core, 64KB pages were a few percent faster, but once I reduced the amount of RAM I found that 4KB pages still worked fine even after it started swapping, but as soon as the 64KB kernel went into swap space, it would start thrashing and never complete the kernel build
<arnd>
or complete it after a day of wearing out the disks, instead of a few minutes of building in RMA
<arnd>
RMA
<arnd>
RAM
<arnd>
sven: ok, good to know, I have to read up on that again
<sven>
i think iommu_dma_map_page uses __iommu_dma_map_swiotlb which uses bounce buffers. and from what i can tell dma_map_page is used for streaming dma
<arnd>
sven: ok, got it, thanks for the pointer
<arnd>
it looks like that was added fairly recently (last November) based on code from the Intel iommu driver.
<arnd>
sven: the dev_is_untrusted() logic in there looks like the exact thing to use for determining whether to set up a linear map or not
<arnd>
for the thunderbolt ports, it would be set using the "external-facing" property, for the wireless device I suppose the driver could set it
<sven>
oh, good point!
<sven>
i think if dev_is_untrested is set it defaults to DMA domains anyway
<sven>
*dev_is_untrusted
<sven>
yup, in iommu_get_def_domain_type there is a check before it calls to the iommu driver to get the default domain: if (dev_is_pci(dev) && to_pci_dev(dev)->untrusted) return IOMMU_DOMAIN_DMA;
<sven>
so if something is untrusted the dart driver will never see a IOMMU_DOMAIN_IDENTITY unless the user forces it
<arnd>
That sounds exactly like what we want, though I'm not sure it would work if the wireless driver sets the flag from the probe function, that is probably too late.
<sven>
true
malvo has joined #asahi
<sven>
hm, but i think we'll somehow need the wireless device in the dt anyway since it needs that special poweron sequence
malvo has left #asahi [#asahi]
<arnd>
sven: right, that would work, we could just put the "external-facing" property on the bridge node, but that feels wrong, as what we really want is to mark that whole class of broadcom wireless devices (or maybe any wireless network device?) as untrusted, regardless of what machine they are used in
bisko has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
EER has joined #asahi
<kettenis>
sven: did you write down your findings about subpage protection somewhere?
<sven>
no, because i'm not 100% sure how it works
<sven>
iirc the region was encoded in the top 24 or so bits of the PTE and you had to clear bit 0 or 1 to enable it
<sven>
hrm, can't find my old code anymore
<Emantor>
sven: The pgsize calculation in the dart driver looks wrong, it's doing a 0x4 > PAGE_SIZE comparison for the USB darts. This way the force_bypass parameter will never work.
<Emantor>
You probably wanted to shift 4096 instead of 1?
<sven>
shift?
<sven>
it's doing 0x4000 > PAGE_SIZE
<sven>
or am i missing something?
<Emantor>
field get returns 0x2 under barebox, i guess the kernel is the same?
<sven>
huh, that's weird
<sven>
it should be 0xe
<Emantor>
Dart Params is:
<Emantor>
16 0 24 12 0 24 20 16 12 8 4 0
<Emantor>
49283072 0x2f00000 0274000000 0b10111100000000000000000000 = 47.00 iM
<sven>
uh
<sven>
i don't understand that fomat
<sven>
dart_params[0] = 0x2ed01010 for the display dart here
<Emantor>
hu, wierd.
<sven>
read32(0x382f00000) = 0x8ee01020
<sven>
that should be the usb dart
<Emantor>
Yeah, there is something broken on my end.
<Emantor>
Argh, wrong regs >_>
<Emantor>
Great, now that the dart driver works by enabling bypass mode, gadget mode receives requests. Sorry for the noise.
<pipcet[m]>
gadget mode in barebox Just Works if you start it from linux :-)
<sven>
same if you start if from m1n1 with my pr *shrug*
<sven>
host mode should just work as well there
<pipcet[m]>
cool
<sven>
turns out you don't actually need a different phy init sequence for gadget/host mode (at least for usb2)
<pipcet[m]>
so switching between host and gadget modes works now?
<jannau>
no new hardware, haha
<sven>
i didn't try switching but the same phy init pokes i have work for either mode without an additional phy driver in linux
<sven>
oh, when you try to switch you'll run into the "using the same port twice isn't supported" issue :D
<sven>
but once we figure that out mode switching should just work as well
<pipcet[m]>
sven: we can just reset the port once every few seconds except when it's connected :-)
<sven>
yeah, no
<sven>
you also only need to reset once
<sven>
so if you want to hack it just spawn a thread that polls and checks if it's connected and if not resets it once
<pipcet[m]>
can't you fix it instead? pretty please?
<sven>
eventually, yeah
<sven>
i first want to get the DART driver and the clocks upstreamed though
<sven>
if i wanted to just get something somehow working we would have a kernel+distro hacked together a few months ago already ;)
<jannau>
I've started working on the gpio driver btw
<svenpeter[m]>
Nice!
<pipcet[m]>
cool
<svenpeter[m]>
You’ve probably seen kettenis’ bindings haven’t you?
<svenpeter[m]>
I think they’ve already been reviewed by the maintainers
<kettenis>
yes, the bindings are going in via the pinctrl tree
Andalu30 has joined #asahi
<Emantor>
pipcet[m]: chainloading the generic 2nd dt image for barebox fails for me, HV works obviously. But the gadget still spews lots of errors.
* Emantor
wishes for a USB PD board.
<jannau>
yes, using the v3 patches as base
<pipcet[m]>
emantor: silly question, but you're passing an appropriate device tree?
<Emantor>
pipcet[m]: I'm passing m1n1 with bb & dt concatenated. So DT should be fine.
<Emantor>
no, wait
<Emantor>
ye, doesn't matter. Still get an apple logo screen chainloading m1n1 with a barebox payload.
<pipcet[m]>
the frame buffer is black on black unless you apply the patch or hack the DT to claim x8r8g8b8 layout.
<Emantor>
I know, this is with your patch applied on top.
<pipcet[m]>
are you running m1n1 without a logo? I'm confused by the apple logo
<Emantor>
You are not alone. No m1n1 is running with a logo and correctly sets up the asahi logo.
<pipcet[m]>
...and then it goes back to the apple logo somehow?
<Emantor>
yep, white apple logo smack down in the middle.
<Emantor>
Running under the hypervisor gives me nice simplefb output with debug apple logo. Where does that even come from?
<Emantor>
debug apple logo == color stripped apple logo.
<svenpeter[m]>
That’s an Easter egg inside mini 😅
<VinDuv>
m1n1 stores the original logo and restores it during fb_shutdown
<VinDuv>
(and changes it if fb_improve_logo is called)
<pipcet[m]>
oh. doh.
<pipcet[m]>
(...my excuse is I didn't get much sleep since I tried to get nvme to work in barebox)
Erus_Iluvatar has quit [Quit: Erus_Iluvatar]
Erus_Iluvatar has joined #asahi
klltkr has joined #asahi
erincandescent_ has quit [Remote host closed the connection]
erincandescent has joined #asahi
klltkr has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
klltkr has joined #asahi
klltkr has quit []
Andalu30 has quit [Quit: Konversation terminated!]
klltkr has joined #asahi
klltkr has quit []
klltkr has joined #asahi
klltkr has quit []
klltkr has joined #asahi
klltkr has quit []
<pipcet[m]>
emantor: do you happen to know whether nvme support in barebox works on other machines? I got something to work but I had to touch the generic code...
<Emantor>
pipcet[m]: Yep, should work on boards with an nvme slot, which are the zii imx8mq boards AFAIK.
VinDuv has quit [Quit: Leaving.]
choozy has joined #asahi
pugguu has quit [Ping timeout: 480 seconds]
<nico_32>
speaking of ubuntu and multiple kernel
<nico_32>
they ship a kernel variant just for microsoft azure