marcan changed the topic of #asahi to: Asahi Linux: porting Linux to Apple Silicon macs | General project discussion | GitHub: https://alx.sh/g | Wiki: https://alx.sh/w | Topics: #asahi-dev #asahi-re #asahi-gpu #asahi-offtopic | Keep things on topic | Logs: https://alx.sh/l/asahi
<jeff_miw>
no uart cable with all the usb pd/vdm stuff ... but I can work how to get one if mandatory. no problem to read the cursed specs :)
<jeff_miw>
let me know once you have something that makes sense to test
<sven>
nah, no need to get one. i hope that i'll be able to finish at least an initial version of the serial part this week. after that most people probably won't need the pd uart cable anymore
<sven>
i'll let you know once i have something that is more useful than just printing what commands it receives over uart :)
jeff_miw has quit [Ping timeout: 240 seconds]
VinDuv has quit [Quit: Leaving.]
<marcan>
sven: it shouldn't be too hard to do CDC serial AIUI
<sven>
yeah, i think once i have enumerating working CDC serial shouldn't be much more complicated
vimal has quit [Ping timeout: 260 seconds]
raster has joined #asahi
mxw39 has quit [Quit: Konversation terminated!]
vimal has joined #asahi
josiahmendes[m] has quit [Quit: Idle for 30+ days]
dyniec[m] has quit [Quit: Idle for 30+ days]
mofux[m] has quit [Quit: Idle for 30+ days]
<maz>
marcan: finally ironed out the annoying timer issue over the weekend, though the result isn't pretty at all. guests now work reliably.
<maz>
marcan: I'll explore a solution that doesn't involve the magic timer LR though, as it prevents the emulation of some GICv3 features Linux relies on.
mofux[m] has joined #asahi
zkrx has quit [Ping timeout: 245 seconds]
<kettenis>
sven: TI probably has slightly different glue logic for the dwc3 controller
<kettenis>
so not everything in that document will apply
<sven>
yeah, especially the PHY doesn't apply at all.
<sven>
but the "USB Device (Peripheral) Programming Model" should be close enough
<sven>
if I do the phy init manually from m1n1 and use my dart driver in linux their dwc3 driver also works without any modifications in gadget mode fwiw
<j`ey>
their = linux or?
<sven>
yes
<kettenis>
you don't have to muck around with the i2c type-c controller?
<sven>
not for gadget mode
<sven>
but I believe that's unfortunately required for host mode
<kettenis>
yeah, my attempts to use host mode failed to detect any devices
<sven>
yeah, same here
<kettenis>
no port status changes
<sven>
exactly
<kettenis>
that's when I switched to looking at pcie
<sven>
:-)
<sven>
I then switched to writing usb code for m1n1 because I was tired to wait for my kernels to load over uart
<spikebike>
55dddd:LK;'
zkrx has joined #asahi
<marcan>
maz: awesome! what's the exact issue?
<marcan>
I think qemu on macos works reliably without that LR stuff, only mask bits for the two timers, though I'm not entirely sure
<marcan>
sven: the ports won't go into DFP mode without poking the type C controller
<marcan>
same reason you need to be DFP for serial in m1n1
<marcan>
the controller doesn't have DFP negotiation enabled by default, so if you present to it as an UFP it just does nothing until macos boots
<marcan>
I bet it's not too hard of a dance though, and those things are mostly documented
<marcan>
the linux driver for it should mostly just work
<marcan>
but that does depend on i2c
<marcan>
which I guess is a very good reason to make that my next target
<marcan>
(need to poke around the pasemi i2c driver)
Jangal has quit [Remote host closed the connection]
Janhouse has joined #asahi
<maz>
marcan: the main problem is that you can't inject a timer interrupt using that magic register. the Pending state is only driven by the timer itself, and only the Active state is writable.
<maz>
marcan: so although we can save/restore the timer state correctly, including the GIC Active state, we can't pend an interrupt from SW, and Linux relies on that for interrupt resending.
<marcan>
ah, so you can save/restore the timer counter state, but not the pending bit?
<maz>
marcan: the GICv3 architecture mandates that you can write to the pending state and generate the corresponding interrupt, so that's pretty annoying.
<marcan>
could you force the issue by, say, setting it up to count down 0 ticks?
<maz>
marcan: you don't need to save the pending bit itself. the state of the timer is enough to regenerate it.
<marcan>
but wouldn't that be enough to inject the FIQ? AIUI it's level-triggered
<maz>
setting TVAL to 0 would be observable by the guest, so that's a bit nono.
<marcan>
oh wait, you mean in that LR
<marcan>
I was talking about the timer IRQ flag itself in the control register
<marcan>
yeah. I noticed only one of the bits is writable in the LR
<maz>
so short of being able to set the Pending bit in the LR, we can't actually emulate the GICR_ISPENDING register for the virtual timer PPI.
<maz>
which is why I'm going to try and switch to just masking/unmasking the interrupt.
<maz>
and rely on a normal LR for injection.
<marcan>
this is for emulating a normal GIC system, not the FIQ stuff, right?
<maz>
we'll get a performance degradation because of the exit on EOI.
<maz>
I'm only concerved with GIC emulation, because that's the only thing KVM supports.
<marcan>
right
<maz>
in the kernel, that is.
<maz>
you can obviously plug a userspace interrupt controller.
<marcan>
so I actually don't know at all how the timers interact with the vGIC stuff
<marcan>
or that magic LR register for that matter
<marcan>
so I need a bit of context :)
<marcan>
is that magic LR connected to the vGIC stuff?
<marcan>
oh, I guess what that HACR bit is redirect the vtimer from FIQ to that LR which then interacts with vGIC?
<marcan>
*bit does
<maz>
yes. the magic LR takes the timer output as its pending bit, and exposes the resulting interrupt in the guest via ICV_IAR*_EL1.
<marcan>
makes sense
<marcan>
and what you're saying is we can't make the timer drive a pending state?
<maz>
it's a really nice shortcut in the sense that it doesn't generate an exit from the VM when the timer fires.
vimal has quit [Quit: Leaving]
<maz>
because that pending state is *only* drive by the timer, SW can't inject an interrupt in that LR despite the timer not firing, which the GICv3 architecture allows.
<maz>
driven*
<marcan>
how would that be used? wouldn't the guest always expect the timer to be pending when it receives an IRQ?
<maz>
in the case of a level interrupt, that's not a huge problem. it's more for edge interrupt that it is problematic.
<marcan>
right, but these timers are level, right?
<maz>
but in general writing to GICx_ISPENDING must generate an interrupt, full stop.
<maz>
and we can't do that for the timer.
<marcan>
oh, you mean if the guest writes that
<maz>
yup.
<marcan>
argh, okay
<maz>
it really is a corner case, but deviating from the architecture is in general a very bad idea.
<marcan>
can we use a regular LR for that?
<maz>
yup, at the expense of using mask/unmask to prevent the timer from firing.
<maz>
which we need to do anyway for the physical timer, which doesn't have a magic LR.
<marcan>
right
<maz>
the main disadvantage is that you need to force an exit whenever the guest EOI a timer interrupt in order to unmask the physical interrupt.
<marcan>
could we use both, somehow? use a regular LR to inject pending states if the guest asks us to, otherwise let the timer use the shortcut LR
<maz>
which we don't have to do on normal GIC implementations, as the HW knows how to link the virtual interrupt to the physical one.
<maz>
marcan: it is illegal to have two LRs with the same INTID...
<marcan>
yes, but a lot of things are illegal about this architecture already...
<marcan>
worth trying it :)
<marcan>
I mean that magic register is already way out in IMP-DEF territory anyway
<maz>
sure, but if I can get decent perf with just mask/unmask, I'd be happy to just ditch the IMPDEF stuff.
<marcan>
sure, works for me
<maz>
becase this is a major hack in the vgic implementation, which is already a very complicated beast.
<marcan>
especially for a first implementation
<marcan>
we can worry about micro-optimizing this later if we need to
<marcan>
when we have a use case
<maz>
yup.
<sven>
marcan: nice, I'm pretty sure usb host most is just going to work as well then once you have that i2c figured out
klaus has joined #asahi
jaalsa has quit [Quit: jaalsa]
jaalsa has joined #asahi
raster has quit [Quit: Gettin' stinky!]
raster has joined #asahi
jeff_miw has joined #asahi
nmg_ has quit [Ping timeout: 246 seconds]
<jeff_miw>
I tried to mess around to understand how macOS is managing the USB port but I failed miserably to have the boot/kernel to take my kext :(. if anyone has a good pointer to replace an existing original macOS by one hacked, I'm interested :)
nmg has joined #asahi
<sven>
there are really three components to it: That TI chip connected over i2c to select the mode (DFP,UFP), the PHY (apple calls this one "ATC" I think, essentially magic register pokes required before accessing the controller), and this designware did controller (xhci + configuration for host mode, custom stuff for device mode)
<sven>
part three is well documented if you just look hard enough. part one is mostly documented as well I think (except for the i2c part) and part 2 is completely custom apple stuff afaict
jeff_miw has quit [Ping timeout: 240 seconds]
<sven>
*designware drd
raster has quit [Quit: Gettin' stinky!]
<arnd>
maz: how does qemu/hvf do it?
<arnd>
does it need a similar hack?
tomtastic_ has joined #asahi
tomtastic has quit [Ping timeout: 240 seconds]
Spiffy has quit [Quit: BRB]
Spiffy has joined #asahi
<maz>
arnd: I have no idea, and to be honest I don't really want to know what macos does...
<maz>
arnd: from what I understand, HVF gives you a prety high-level interface to the vcpu, and can do a lot of stuff in the hypervisor layer.
<arnd>
ok, so it's not just emulated in qemu then
<maz>
arnd: I doubt it. performance would be absolutely terrible, similar to what we end-up with running KVM on a RPi3...
<marcan>
arnd, maz: as far as I know hypervisor.framework only deals with IRQ/FIQ, it does not do vGIC
<marcan>
so it should all be emulated in userspace
<marcan>
all you get is management of the pending IRQ state and the vtimer mask
<marcan>
"After receiving a HV_EXIT_REASON_VTIMER_ACTIVATED exit reason, the caller of hv_vcpu_run() needs to make the interrupts to the VTimer pending in the guest’s virtual interrupt controller."
odmir has joined #asahi
lethalbit has quit [Read error: Connection reset by peer]
lethalbit has joined #asahi
vimal has joined #asahi
<maz>
marcan: it has to do *some* vgic in the kernel, as the LRs cannot be populated from EL0. from what I see, their interface is exactly the same as KVM, except for the timer (and I don't really understand why).
<marcan>
maz: I think they don't do vgic at all, and it's all emulated in userspace?
<maz>
marcan: to do the GIC in userspace, you'd need to pass the trapping of all ICV_* regs to userspace.
<maz>
plus the distributor and RD MMIO as well.
<maz>
are they actually doing that?
<maz>
that'd seem... silly.
<maz>
why would they build a vgic the first place if they aren't using it...
<marcan>
because this launch was very, *very* obviously rushed
<marcan>
and they probably never got around to implementing it
<marcan>
but there is nothing on IRQs in the API other than "IRQ, FIQ"
<maz>
they don't have any exit reason for a trapped sysreg.
<maz>
I think you are confusing two different things: injecting a IRQ/FIQ directly, and injecting a GIC interrupt.
<marcan>
probably HV_EXIT_REASON_EXCEPTION?
<maz>
hv_vcpu_set_pending_interrupt clearly injects a GIC interrupt, because there is an interrupt value provided to it.
<maz>
IRQ?FIQ is just the grouping here.
<marcan>
ah, I think you're right
<maz>
(grouping in the GIC sense)
<marcan>
I skimmed too quickly there
<maz>
I think they have the distributor in userspace, while KVM has it in the kernel.
<marcan>
yeah
<marcan>
and in that case they probably don't use that magic LR
<maz>
and that explains why userspace needs to mediate the vtimer as well.
<marcan>
wait, no, are you sure?
<marcan>
the structure is called hv_interrupt_type_t
<maz>
then I don't understand how this stuff works.
<marcan>
I think all the registers trap...
<marcan>
agraf would know :)
<maz>
great for performance... I'll stay with my in-kernel, non-trapping handling, thank you very much.,
<marcan>
maz: I'm really looking forward to this, running a few benchmarks side by side on qemu/macos vs qemu/kvm will make for good clickbait ;)
<maz>
marcan: qemu needs fixing, there is a stupid but with the reduce IPA space.
<maz>
reduced*
<marcan>
ah, heh
<maz>
kvmtool and crossvm are equially broken.
<j`ey>
qemu on linux on m1 you mean?
<maz>
yup.
raster has joined #asahi
<agraf>
maz: I'm not aware of any hw backed GICv3 MSRs, but I haven't looked hard from EL2 either. HVF definitely just traps and delegates emulation to EL0 for everything. The only remotely irq related thing that's handled in-kernel is the arch timer, and that also just tells user space when it crossed cval.
<maz>
agraf: KVM directly uses all the vgic registers without any trapping.
<maz>
agraf: all the standard GICv3 stuff is there and seems functionnal.
<agraf>
maz: oh, so they actually implemented all of it?
<agraf>
That's funny - in a way :)
<maz>
agraf: just enough to have a guest that doesn't suck too much.
<maz>
agraf: they of course miss all the linked deactivation, but hey, sub-standard HW... ;-)
<agraf>
maz: I'm definitely eager to see some comparison on native vgic vs the el0 loop for real world use cases. Probably has quite a nasty impact on IPI heavy bits
<agraf>
maz: macos doesn't use any of the gic parts, so expect pitfalls as large as TX1 ;)
<agraf>
I'm fairly confident it's not fully verified ip
<maz>
agraf: running hackbench (as IPI-heavy as it gets) in a guest, I get a 6/7% overhead.
<maz>
overhead compared to running on bare-metal.
<maz>
agraf: as for bugs, I'm planning to taint the kernel with CPU_OUT_OF_SPEC at KVM-init time.
Hetflik[m] has joined #asahi
never_released has quit [Changing host]
never_released has joined #asahi
<marcan>
ha
<marcan>
OUT_OF_SPEC?
<marcan>
what spec?
<marcan>
:D
<never_released>
maz: so there is a GICv3?
<never_released>
in addition to AIC?
ldhacker[m] has quit [Quit: Idle for 30+ days]
<maz>
never_released: no. what's available is the ICH_*_EL2 and the ICV__EL1 registerse.
<maz>
registers*
<maz>
never_released: you get enough to *virtualise* a GICv3.
<never_released>
makes sense
<never_released>
maz: pretty odd that macOS doesn't use that at all
<never_released>
and just traps all the regs to the VMM through Hypervisor.framework
<maz>
never_released: as agraf said, it's possible that it is totally broken.
<maz>
never_released: but as long as the traps work, we can deal with that in the kernel, where we have all the required emulation already.
<marcan>
I still bet it works ~fine and they just didn't make shipping for macos
<marcan>
I mean ffs, the version of macos that shipped on these things had some hideous brokenness in the boot process; everyone was screaming "update before you even dare do anything"
marvin24 has quit [Ping timeout: 240 seconds]
marvin24 has joined #asahi
TheJollyRoger has quit [Quit: TheJollyRoger]
VinDuv has joined #asahi
mxw39 has joined #asahi
aratuk has joined #asahi
aratuk has quit [Remote host closed the connection]
<marcan>
writing blog posts takes a while... I'm 3500 words into the progress report and I haven't even gotten to the linux kernel yet
<marcan>
hopefully I can mostly wrap that up tomorrow
KindTwo has joined #asahi
KindOne has quit [Ping timeout: 272 seconds]
KindTwo is now known as KindOne
TheJollyRoger has joined #asahi
<jn__>
wow, excellent
klaus has quit [Ping timeout: 272 seconds]
choozy has joined #asahi
aratuk has joined #asahi
aratuk has quit [Ping timeout: 245 seconds]
raster has quit [Quit: Gettin' stinky!]
amw has joined #asahi
marvin24 has quit [Ping timeout: 240 seconds]
marvin24 has joined #asahi
eta has quit [Ping timeout: 245 seconds]
artemist has quit [Ping timeout: 240 seconds]
VinDuv has quit [Quit: Leaving.]
eta has joined #asahi
Guest56 has joined #asahi
choozy has quit [Remote host closed the connection]