<jernej>
linkmauve: header parsing in kernel won't be accepted for mainline kernel
<jernej>
all data must come from controls
Daanct12 has joined #cedrus
Daaanct12 has joined #cedrus
Daanct12 has quit [Ping timeout: 480 seconds]
<linkmauve>
jernej, the kernel already has include/media/v4l2-jpeg.h for JPEG header parsing.
<linkmauve>
Used by the coda and imx platforms.
<linkmauve>
But it makes less sense to do that in the request API I think.
Daaanct12 is now known as Daanct12
<linkmauve>
Ugh, my phone just rebooted once I ^C’d this command: % gst-launch-1.0 filesrc location=Videos/vp8.webm ! matroskademux ! v4l2slvp8dec ! glimagesink
<linkmauve>
It otherwise decodes it almost correctly, just there is a green line at the bottom, probably some incorrectly-negociated buffer size somewhere.
<linkmauve>
And I can reproduce the reboot.
<linkmauve>
Is Gstreamer the easiest userland to work on that, or is there something a bit more self-contained?
<linkmauve>
I have a crate for dealing with the media and request APIs, but so far no video API support so I can’t really use it to test decoding.
<linkmauve>
Ok, not just if I ^C, also if the video finishes properly.
<linkmauve>
And besides the reboot and the green line, there is also a weird chroma drift from the top-left towards the bottom-right, which gets reinitialised on keyframes or strong movement.
<linkmauve>
And it also happens with H.264.
<linkmauve>
(I’m on a master kernel built earlier, I could check on 5.16 if it’s a regression.)
<linkmauve>
With H.264 I also get this warning: imported bo size is smaller than expected: 462848 (BO) < 471040 (expected)
<linkmauve>
No, same behaviour on 5.16.13…
<linkmauve>
Also, when using waylandsink instead of glimagesink, I get this: (gst-launch-1.0:2118):
<linkmauve>
ERROR: from element /GstPipeline:pipeline0/v4l2slvp8dec:v4l2slvp8dec0: Unsupported pixel format
<linkmauve>
With videoconvert in the middle it does play, but with a very green tint!
<linkmauve>
Of course, waylandsink would have had to support NV12_32L32, which is probably specific to this SoC.
Daanct12 has quit [Quit: Quit]
<jernej>
linkmauve: afaik coda and imx platforms use stateful driver, where parsing is done by coprocessor, not kernel driver
<jernej>
it was agreed a while back that request api drivers won't do any kernel parsing
<jernej>
why don't you use normal NV12?
<jernej>
last time I tested gstreamer h264 support on H6, it rendered correctly directly on screen, but that was without any desktop environment
<linkmauve>
jernej, not sure, it was automatically negociated by Gstreamer, I’m just looking at the pipeline SVG it produced (before crashing my phone…).
<linkmauve>
Now I’m testing on a SBC, which doesn’t crash but has an A20 instead of an A64.
<linkmauve>
From what I read in some cedrus file, the A20 only supports 32L32.
<linkmauve>
(I assume this is what CEDRUS_CAPABILITY_UNTILED means.)
<jernej>
true
<jernej>
I thought you're using something newer
<jernej>
if you're running gstreamer without desktop environment and using kmssink, it should work and you don't even need videoconvert
<linkmauve>
My second screen is busted, so I’m relying on waypipe for now. :(
<jernej>
at the end, you can always decode to images and check that
<linkmauve>
Do you have any sample code for that?
<linkmauve>
I’m still in the middle of adding video API support to my crate, but I won’t know whether this userland code isn’t working or if it’s the kernel driver, or even the hardware. :D
aedancullen has quit []
aedancullen has joined #cedrus
aedancullen has quit []
aedancullen has joined #cedrus
<jernej>
no, I'm more of a ffmpeg user
<jernej>
but there is nothing in upstream, just collection of patches
<jernej>
nothing for JPEG request api though
<linkmauve>
What is holding up upstreaming these patches?
<jernej>
time and motivation
<jernej>
2 rounds were already done, but original author got busy IRL
<jernej>
now that RPi also uses request API for HEVC, there are additional improvements, which nobody make sure yet that works for all
<jernej>
you can volunteer :D
<linkmauve>
:/
<linkmauve>
I don’t have any Raspberry device to test against though, only a bunch of Allwinner, one Amlogic, and one Nintendo.
<linkmauve>
And a Qualcomm which doesn’t start any more, I’d like to figure out why someday.
<jernej>
it's relevant only for RPi4
<jernej>
older use SW HEVC decoding
<linkmauve>
I could try to shepherd this series to upstream, this wouldn’t be my first time. :p
<linkmauve>
Btw, thanks you for making me figure out that kernel dev isn’t harder than userland dev!
<linkmauve>
I’ve been steadily upstreaming the Wii U port this past year, writing drivers as it goes. :)
<jernej>
sure, I'm just not sure how I helped you to come to this conclusion :)
<linkmauve>
It was about the VP8 decoder I helped fix for 5.11. ^^
<jernej>
ah
<jernej>
VP8 isn't best representation of request api driver
<linkmauve>
It’s in a state of “at least it boots”, and completely based off the MPEG2 code.
<linkmauve>
Would it be ok to basically require userland to pass a struct v4l2_jpeg_header as the ctrl, btw?
<jernej>
looking at current linux code, there is no jpeg request api decoder
<linkmauve>
Yes, that’s what I’m trying to add.
<jernej>
so you have to invent some things first, like format
<jernej>
V4L2_PIX_FMT_JPEG is full image, you should add something like V4L2_PIX_FMT_JPEG_PARSED
<jernej>
although probably more sensible
<jernej>
but yeah, if aforementioned structure contains everything you need, it can be used as control
<jernej>
uh, no
<jernej>
it contains pointers, that a no-go
<ndufresne>
"linkmauve> I have a crate for dealing with the media and request APIs, " ineresting, I have a colleague working on VA decoder for cross-vm, virtio-gpu, and V4L2 Stateless CODEC support (I hate calling is request), is coming next
<ndufresne>
He's adding crate for whatever is missing or not good enough for parsing, but also for state manangement
<ndufresne>
jernej: oh, and yes, there is exception for jpeg stateless HW in few places, cause it was "simple and safe enough"
<ndufresne>
they don't call it stateless, since that make no sense, JPEG encoder are always stateless
<ndufresne>
they call it raw encoder, or raw decoder (without headers)
<jernej>
ok, then linkmauve, your approach should work
<jernej>
just make sure you properly set start of bitstream
<jernej>
which should be right after the header
<jernej>
ndufresne: is there any userspace app which knows how to use v4l2 jpeg decoder?
<ndufresne>
jernej: chromium, gstreamer, not sure why ffmpeg stateful v4l2 could not support that
<ndufresne>
it all statefull, so you collect an entire jpeg in 1 buffer, and give it to the driver
<ndufresne>
as its stateless, if you picked the buffer size wrong, you just reset the thing and realloc
<ndufresne>
live is so easy with still pictures
<ndufresne>
* life
<ndufresne>
of course, it not ideal if you HW support progressive load, but its not a trendy these days
<ndufresne>
these are mostly use to handle mjpeg, normal jpeg but produce by camera to workaround USB2 bandwidth
<linkmauve>
jernej, start of bitstream, do you mean the offset I’ve (so far) left set to 0?
<linkmauve>
ndufresne, would you prefer me to publish what I already have, or to add it to their existing crate?
<linkmauve>
would they* prefer ^^
<linkmauve>
So far it’s just a repository on my SSD.
<ndufresne>
linkmauve: perhaps ping dwlsalmeida on #gstreamer, he's the colleague I'm referering to
<linkmauve>
Ack. :)
<jernej>
yes, I mean offset
<ndufresne>
not sure were his stuff goes tbf, but he said that he got VP8/VP9/AV1 ready, and will start H.264
<jernej>
but you also didn't select jpeg mode
<ndufresne>
offset to the raw jpeg start, skipping the header right ?
<jernej>
yes
<ndufresne>
and I wouldn't be surprise this is what pH5 helper for coda does
<jernej>
btw, what is your source for register documentation? old CedarC source?
<linkmauve>
jernej, jpeg mode where? In the cedrus registers?
<jernej>
yes
<linkmauve>
cedrus_engine_enable(ctx, CEDRUS_CODEC_JPEG); does that I think.
<jernej>
that's engine
<jernej>
which can decode mpeg1/2/4, vp6.2 and jpeg
<jernej>
so you have to tell which codec you have in mind exactly
<linkmauve>
In the proof of concept code I used as an example, set_format() was a thing which wrote to one byte of the trigger register, so far I hoped writing to all four bytes as an uint32_t would do the same.
<linkmauve>
They do writeb(0x0e, ve_regs + VE_MPEG_TRIGGER);
<linkmauve>
Oh, I’m wrong, I still had VE_DEC_MPEG_TRIGGER_MPEG2 from the MPEG2 code.
<linkmauve>
Will change that asap, thanks. :)
<linkmauve>
“22:13:45 ndufresne> jernej: chromium, gstreamer, not sure why ffmpeg stateful v4l2 could not support that”, how do you do that with Gstreamer? I couldn’t find a v4l2sljpegdec element anywhere.
<jernej>
you set VE_DEC_MPEG_TRIGGER_MPEG2 instead of VE_DEC_MPEG_TRIGGER_JPEG
<linkmauve>
That’s fixed now, thanks. :)
<linkmauve>
The offset as well.
<jernej>
btw, don't use static variables inside functions
<linkmauve>
I went for a static buffer in order to avoid blowing up the stack, the compiler was configured to warn for any function using more than 1024 bytes of stack apparently.
<linkmauve>
Should I make it a global instead?
<jernej>
no, static buffer is same as global
<jernej>
and neither is acceptable for kernel
<jernej>
allocate it via kmalloc and store pointer in jpeg context
<linkmauve>
Ok.
<jernej>
and maybe it would be possible to avoid it with some code reorganization
<linkmauve>
I could do that, yeah.
<jernej>
ndufresne: which helper do you have in mind?
<linkmauve>
I don’t understand the data we have to put in said buffer at all.
<linkmauve>
Like, the sum *= 2; is black magic to me.
<linkmauve>
Or why we never have to put the last number of the huffman tables.
<ndufresne>
jernej: ./drivers/media/v4l2-core/v4l2-jpeg.c (but I never looked at the one), I mostly play with the h264 and vp9 one, which are not about parsing
<jernej>
I have no idea about actual register values
<linkmauve>
I’ll just leave the values as they are, and try to avoid the buffer altogether.
<linkmauve>
I think I can reduce the memory usage from 2048 to 64 bytes.