08:23 <mwalle> lynxeye: sorry I had to leave yesterday. Yes I'm aware of the PTA which in turn selects the MTLB base address

08:24 <mwalle> lynxeye: but the driver sets the MTLB base address in the PTA entry as well as in etnaviv_buffer_config_mmuv2()

08:31 <lynxeye> mwalle: Ah, yea. The mtlb address in the config reg is simply ignored by GPUs with PTA support. I see how this is a bit confusing.

08:33 <mwalle> lynxeye: ah ok, thanks

08:35 <mwalle> lynxeye: one thing which is odd, the base address of the PTA can be 40bit, the PTA can point to a 40bit mtlb base address but an mtlb entry cannot point to a 40bit address (but the stlb entry can result in a 40bit target address again..)

08:35 <lynxeye> mwalle: You see how we skip adding the mtlb address in the flush path in etnaviv_buffer_queue? We could do the same thing in etnaviv_buffer_config_mmuv2 to make things a bit more clear, but it doesn't matter to the HW.

08:37 <lynxeye> mwalle: I'm not sure if the mtlb 32bit limit is intentional, or simply a oversight on my side. Have you checked the vivante kernel driver? I would guess that you can just fold the 40bit address for the mtlb entry the same way as is done for the stlb entries.

08:37 <mwalle> lynxeye: yes I've checked the vivante kernel driver, and i also played with the tlb stuff quite a bit

08:39 <mwalle> it also seems that (sometimes? always) bit 4 of the MTLB is a "secure" bit, so just folding the upper 8 bits to 11:4 doesn't work

08:40 <lynxeye> That's odd... So maybe we need to limit the coherent dma mask to 32bit after all. That should not really hurt, as the only coherent dma allocations done by etnaviv are for the command stream buffer and the pagetables.

08:40 <mwalle> let me see if i can find that piece of code..

08:43 <mwalle> lynxeye: at least that is for the "secure world mode": https://github.com/Freescale/kernel-module-imx-gpu-viv/blob/master/kernel-module-imx-gpu-viv-src/hal/security_v1/gc_hal_ta_mmu.c#L436

08:45 <mwalle> but I've tried to insert upper bits of a 40bit address of the stlb into the mtlb entry and it doesn't work. (I've traced that with an IOMMU exception, expecting that if I change the address, it will trigger a SMMU exception, which it does for both the MTLB entry as well as a PTA entry)

08:45 <mwalle> but unfortunantly not for a "wrong" stlb address in the MTLB

09:08 <lynxeye> mwalle: Yea, that's also how I read the vivante kernel driver code. The vivante driver also adds GFP_DMA32 to the stlb page allocations and I see no code to add the upper 8bits in the mtlb. So seems the stlb part of the pagetables must be below the 4GB mark...

09:22 <mwalle> lynxeye: yep, but if theres an IOMMU in between this GFP_DMA32 doesn't really help (if i'm not missing anything)

09:23 <mwalle> the iommu driver just looks at the dma_mask to find a free iova

09:24 <lynxeye> mwalle: Huh? Even with an iommu in between dma_coherent allocations should take the dma_coherent_mask into account.

09:24 <lynxeye> GFP_DMA32 is useless with coherent allocations, as the coherent alloc paths ignore it in favor of the coherent dma mask.

09:27 <lynxeye> mwalle: Currently etnaviv set both dma_mask and coherent_dma_mask to 40bits. You need to split this and only set coherent_dma_mask to 32bits.

09:27 <mwalle> lynxeye: yes dma_coherent_mask is taken into account, but how can i tell linux to get a page <32bit if my dma_coherent_mask is 40bit, that isn't possible I guess

09:28 <lynxeye> mwalle: No, just reduce the coherent mask to 32bits.

09:29 <lynxeye> We only allocate page tables and command stream buffers from coherent memory, so it shouldn't hurt to always put them into the 32bit area.

09:34 <mwalle> lynxeye: I can try, but to cite arnd:

09:34 <mwalle> arnd | At one point we changed all manual assignments of dma_mask to 'dma_coerce_mask_and_coherent)

09:35 <mwalle> arnd | ' to annotate the fact that this is a bad thing of the driver to do, in the hope that new drivers would not attempt it

09:37 <lynxeye> mwalle: If our requirements for coherent and non-coherent are different I don't see why we shouldn't use the facilities provided by the kernel for this case...

09:40 <mwalle> lynxeye: ok, but it still bothers my why just that one address has to be in the 32 bit range. that doesn't satisfy the sheldon in me *g* its like the hardware is 90% there but stumble on the last 10%

09:42 <lynxeye> mwalle: Hahaha, get used to this feeling. There's a lot of cases where the HW is like 90% there, but then falls over. ;)

09:56 <mwalle> lynxeye: btw if the MMU is disabled, there is an identity mapping, right?

09:57 <mwalle> ie the mmu is just bypassed and but memory access is possible

09:58 <lynxeye> mwalle: Yes, as long as the MMU is disabled all GPU addresses are identity mapped to bus addresses.

10:00 <lynxeye> This is required to start the GPU, as you need to execute a command stream to enable the MMUv2.

10:00 <mwalle> lynxeye: yeah I saw that in the start_fe() function

10:02 <lynxeye> TBH I never checked if all GPU engines can access memory without the MMU being enabled, or if just the FE bypasses the MMU, but I guess they all take the same path.

10:06 <mwalle> lynxeye: btw there is a ACE (whatever that stands for) feature, which when availble can be used to specify the upper 8bits for the mtlb and safe address, which as you mentioned is just used if there is no PTA. Thus not in my case. I have the code though, I'm not sure if it is still intersting or if i should just drop it

10:16 <lynxeye> mwalle: ACE means the GPU supports the AMBA ACE bus standard, which seems to be the same point at which they added the 40bit address support.

10:18 <lynxeye> I'm not sure how much value adding more 40bit support has if we just constrain coherent allocations to 32bit, which seems like the most robust solution to me at the moment.