10:43 <marcan> hm, since someone mentioned ANE again... I'm wondering *how* that would even be implemented?

10:43 <marcan> my hunch is it should be a DRI driver, and I see there's been drama over this already and people aren't doing that because DRI requires open source userspace

10:43 <marcan> (cc arnd)

10:44 <arnd> what is ANE?

10:44 <marcan> the neural engine thing

10:45 <arnd> ah, right. What do we know about the hardware side, and about how applications access it in MacOS?

10:46 <marcan> I haven't looked at it at all, but AIUI it's an ASC with command submission and apparently a DART, so probably less fancy address translation than the GPU

10:46 <marcan> not sure how it does context switching

10:46 <marcan> but it smells like something GPU-shaped to me

10:47 <marcan> oddly enough though, it isn't described as an RTKit ASC in the ADT, so maybe it isn't?

10:47 <arnd> the main question to me is whether it implements a particular set of high-level matrix operations (GEMM, convolution, ...) that can be abstracted using a kernel interface, or if this isx a fully programmable unit that relies on JIT-compiling your ML model into a custom ISA and run it autonomously

10:49 <TheLink> https://github.com/hollance/neural-engine/blob/master/docs/reverse-engineering.md

10:49 <TheLink> perhaps already known

10:51 <marcan> there's definitely at least some finite state machine concept

10:52 <marcan> https://github.com/geohot/tinygrad/tree/master/accel/ane

10:53 <marcan> seems to have a fairly high level interface with program submission, queues, prioritization

10:54 <marcan> https://mrcn.st/p/QQy5O82I this kind of stuff

10:54 <marcan> not sure if that's the AP interface but it sounds like it could be

11:02 <arnd> right, this does sound like an ioctl-type interface to send compiled code to the engine and run that, which is indeed similar to what GPUs would do, and also a bit like what we did a long time ago with spufs

11:04 <arnd> https://lwn.net/Articles/870418/ summarizes the discussion we had at the kernel summit about those devices.

11:09 <arnd> in this case, I don't think the question of open source user space is the main issue. As we won't be running MacOS user space on Linux for this, someone has to complete the reverse-engineering anyway in order to create a new compiler

12:10 <alyssa> arnd: right, if we have any linux support, it'd be an open user space

12:20 <alyssa> marcan: my 2c is to stick it in drivers/gpu/drm and use the standard DRM/GEM interface just like AGX

12:20 <alyssa> but I'm also biased as hell

12:20 <alyssa> "gpu" stands for "generic programming unit" ;-)

12:24 <alyssa> marcan: but do AGX first :>

12:25 <alyssa> and AVD and AVE

12:25 <alyssa> those coprocs actually matter

12:25 <alyssa> I am genuinely unsure why ANE would matter on Linux outside of some niche spaces

12:26 <arnd> The GPU programming model always feels like a layering violation to me, the same way that using DPDK to send network data does, or using a custom interface to a DSP doing video encoding instead of using v4l2. OTOH I have no idea what a good kernel abstraction for machine learning hardware would actually look like, so it's probably the best we can do here

12:27 <TheLink> emulate the cuda api with ane :B

12:27 <alyssa> arnd: nod.

12:27 <sven> just create a new machine learning subsystem!

12:28 <alyssa> arnd: I would also point out the GPU ioctl interfaces are extremely NIH

12:28 <alyssa> but every mature driver by definition is committed to an existing uAPI so there is a strong incentive not to change it.

12:28 <alyssa> krh did a neat PoC of what a "common" interface could look like but. there's no turning the clock back on anything mainline

12:28 <jn> the other AI accelerator driver that i'm aware of lives in drivers/misc/habanalabs, probably with a very custom interface

12:29 <arnd> TheLink: cuda is a user space interface, not kernel level, and their kernel interface is not an abstraction but hardware specific

12:29 <jn> (userspace interface)

12:29 <arnd> one could do something that looks like cudnn or cublas, and build on top of that

12:30 <arnd> which in turn is what apple's Accelerate framework or Intel's OneAPI do as well, but I have not seen anyone do such an abstraction on the kernel to user boundary

12:33 <arnd> alyssa: on a related note, do you think it would be possible to have an OpenCL based BLAS implementation on top of your gallium driver, and use that for machine learning on the GPU instead of the ANE?

12:34 <arnd> I'm thinking of applications that use cublas on nvidia GPUs today, not their higher level interfaces or the tensor cores

12:35 <alyssa> sure, clover provides OpenCL on top of Gallium drivers

12:35 <alyssa> it's not ready for production yet, but it's closer to it than the AGX stack ;-)

12:42 <alyssa> I had a dream about Apple sending me an M1 MBA and M1X MBP respectively. It wasn't a very interesting dream.

13:16 <TheLink> that's the way it should be ... apple sending you devices and the process being totally uninteresting :)

14:10 <chadmed> just reading geohot's description of its data handling, it seems very GPU-

14:10 <chadmed> GPU-like*

14:11 <chadmed> the ANE, that is