<marcan>
pcc: no, it's not about tty cooking stuff, it just outright buffers/does not deliver some USB packets
<marcan>
as a workaround I duplicated the tty kick in the ACM driver with a memory barrier in between and that fixed it
<marcan>
remains to be seen where the real problem lies
<marcan>
could even be an issue in my AIC driver, but then I'd be surprised it doesn't break something else
<marcan>
I suspect it might have something to do with the workqueue getting scheduled before the buffer kick is visible to the other CPU, but I'm not sure if workqueue scheduling is supposed to implicitly have a memory barrier
<marcan>
anyway, too busy with random stuff right now, I'll revisit this one later
the_lanetly_052__ has joined #asahi-dev
<marcan>
actually, this might be AIC...
<marcan>
the GIC driver has this
<marcan>
/*
<marcan>
* Ensure that stores to Normal memory are visible to the
<marcan>
*/
<marcan>
* other CPUs before they observe us issuing the IPI.
<marcan>
dmb(ishst);
<marcan>
maz: I don't think we do anything in AIC to ensure memory ordering of regular stores with IPIs, we only do all the fine grained ordering of the AIC driver's internal stuff
<marcan>
that sounds like a bug...
the_lanetly_052 has quit [Ping timeout: 480 seconds]
<marcan>
ah, but we do have an atomic release which shold be creating this ordering
<marcan>
so unless I got something subtly wrong there, this isn't the bug
<marcan>
back to suspecting the tty layer then
<maz>
marcan: ISTR that we had that discussion at the time, and came to the conclusion that the atomics were enough to create the ordering.
<marcan>
yeah, that rings a bell
<maz>
note that this doesn't by itself order with the sysreg access that generates the IPI, but the control dependency should be enough here.
<maz>
(though it took me some time to convince myself)
<marcan>
yup
<marcan>
the tty layer does some stuff then queues some work in a workqueue, and doing that twice with a barrier in between seems to fix the bug. but I suspect the problem is the stuff, not the workqueue ordering now.
<marcan>
deep in the tty layer bowels there's a ton of atomics/acquire/release nonsense which smells to me like it probably has a bug somewhere
<marcan>
and it wouldn't at all be surprising if it only hits M1s with their massive reorder buffers
<marcan>
at least I have an easy reproducer now (repeatedly readmem()ing a certain size in m1n1 triggers it in seconds)
<marcan>
just need to set up a silly chain of 3 systems when I get home to debug this (host to test, target to run kernels on, second target to run proxyclient against :))
<maz>
yup, doesn't seem unlikely. I've been chasing a similar issue with QEMU's implementation of virtio that goes in the weeds when the machine gets properly hammered.
<marcan>
ouch
<maz>
and only M1 triggers it.
kov has joined #asahi-dev
<maz>
but it is really hard to trigger (8 parallel VMs on a mini, and only one failure per 6 or 7 runs)
<marcan>
ouch, that sounds painful
<marcan>
at least this one is easy
<marcan>
it was killing most of my hypervisor runs...
<maz>
yeah, I'm not actively looking, too many things going on ATM.
<marcan>
the annoying thing was tracing it through the stack, because it would just hang and I was at first convinced it was my fault (e.g. something in the HV speculating into bad memory and crashing)
<maz>
hypervisors are crap. let's go shopping! :D
<marcan>
went through the chain, HV -> usbmon -> cdc-acm driver and now I've arrived at tty as the likely culprit...
<kode54>
annoying M1 virtualization feature Apple will probably never fix
<kode54>
VMs are useless for development if you want to run from the latest beta but want to dev in a VM running stable
<kode54>
can't log into iCloud in a VM
<marcan>
because VMs are insecure, no real SEP/etc
<marcan>
pretty sure that's by design
<kode54>
guess I'll just stop developing for Apple computers
<kode54>
I can develop for Windows in a VM just fine
<marcan>
macos VMs are offtopic though
<kode54>
yeah, I realized this wasn't the offtopic channel
bisko has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]