12:36 <mwalle> after a long time, I'm back at debugging etnaviv on ls1028a. I want to tackle the iommu problem. What I *think* I know is that the GPU wants to write at a (virtual) address which is not mapped by the IOMMU.

12:36 <mwalle> [ 51.705120] arm-smmu 5000000.iommu: Unhandled context fault: fsr=0x402, iova=0xfad00000, fsynr=0x70000, cbfrsynra=0xc47, cb=1

12:37 <mwalle> But I don't even know where to begin to look which part of etnaviv might want to read/write there

12:37 <mwalle> or maybe there are some debugging options which I could enable

12:38 <mwalle> I've enabled the VERB() macro and drm.debug=0x11f already

12:43 <austriancoder> mwalle: I think you want to want to look at the cmd stream that triggers the mmu fault. Use https://github.com/etnaviv/etna-gpu-tools to get a gpu core dump and https://github.com/etnaviv/etna_viv/blob/master/tools/dump_separate_cmdbuf.py .. then you should find the place in the cmd stream that triggers it

12:44 <mwalle> I just noticed that error happens really early: https://0x0.st/-V7Y.txt

12:47 <austriancoder> mwalle: if it happens quickly is a big plus

12:49 <mwalle> austriancoder: there are no command streams at that point, right?

12:50 <austriancoder> mwalle: it looks like there is something executed .. [ 58.103955] cmd 000008c0: 40000007 000010a8

12:51 <mwalle> austriancoder: but there is a fault before that at [ 55.806962]

12:54 <austriancoder> mwalle: ahh I see.. but that not the etnaviv iommu but the arm-smmu 5000000.iommu but that seems to trigger an etnaviv-gpu f0c0000.gpu: AXI bus error

12:55 <mwalle> austriancoder: ah, sorry, yes, I'm talking about the IOMMU of the SoC, i.e. the ARM SMMU

12:56 <austriancoder> my knowledge on your platform is zero (and I do not have any HW) .. so I think I can not help you that much.

12:56 <austriancoder> mwalle: maybe lynxeye can help you out

12:57 <mwalle> the axi bus tries to access address 0xfad00000, but I don't know why

12:58 <mwalle> but if it is that early, I might be lucky and doesn't need to understand how this is all working together. Sorry I've never done anything with mesa or graphics

12:59 <mwalle> I take that there is mesa in userspace which assembles some commands which then are send to the GPU, but I guess userspace doesn't know anything about physical addresses

13:04 <lynxeye> mwalle: The iova you get from the SMMU is the "phys" address of the dma buffers used by the GPU. You should be able to figure out which buffer is causing this issue by dumping the dma allocations in the kernel driver and the mapping of buffers into the GPU mmu.

13:05 <lynxeye> 0xfad00000 at least looks like it's trying to access a real buffer you should be able to find this way and not some random uninitialized state.

13:05 <lynxeye> But then first things first, have you hooked up the GPU DT node with the IOMMU with the correct stream ID?

13:07 <mwalle> lynxeye: yes, I've also change the stream id (in the SoC register) manually, which then results in a different message (unknown stream id or something like that)

13:08 <mwalle> gpu@f0c0000 {

13:08 <mwalle> iommus = <0x0f 0x47>;

13:08 <mwalle> which also corresponds to the cbfrsynra

13:09 <mwalle> and the AMQR register also is set to 0x47

13:09 <lynxeye> mwalle: Oh, one more thing to check: the kernel driver is using the virtual drm device to do the dma_mapping, maybe you need to carry over the iommu attachment from the real GPU device to this virtual device. Otherwise the shmem buffer might not get mapped in the system iommu.

13:10 <mwalle> lynxeye: what is the virtual drm device? are you talking about the kmsro thingy?

13:11 <lynxeye> Hm, no. This should already happen through the of_dma_configure call in etnaviv_init.

13:11 <mwalle> where is the framebuffer allocated? there is one, right?

13:12 <lynxeye> mwalle: All Vivante GPUs in a system are exposed to userspace through a single virtual DRM device. But there can be multiple real GPU devices backing this DRM device.

13:13 <lynxeye> mwalle: Framebuffer memory is allocated from the display output device.

13:16 <lynxeye> mwalle: Normal paged memory for the GPU only buffers is allocated via etnaviv_gem_shmem_get_pages.

13:39 <mwalle> "[ 25.852241] thermal thermal_zone1: core-cluster: critical temperature reached, shutting down" oh i need some proper cooling first *g

13:40 <mwalle> lynxeye: is there alreay a way to dump the dma_mappings (or trace them?). do you happen to know that?

13:47 <lynxeye> mwalle: I don't think so. However just sprinkling some prints in etnaviv_iommu_map should already give you an idea what is being mapped on the GPU side and if the iova you see on the system iommu side is somewhere in there.

13:52 <mwalle> lynxeye: ah there is already that VERB(), which leads to the "map[n]:" prints at the beginning of the debug log above. didn't find this address there

13:52 <mwalle> thus i was asking about the framebuffer

13:54 <lynxeye> mwalle: Framebuffer is just another buffer from the GPU PoV, there is nothing special about it.

13:56 <mwalle> lynxeye: so you mean it should also be mapped.. makes sense

13:57 <mwalle> lynxeye: I have to go now.. thanks for now

13:57 <mwalle> austriancoder: ^

13:57 <mwalle> and for your patience with me :p