Not working with ffmpeg

[strongtz]

First of all, thanks a lot for your great work!
I tried this on rk3568 with mainline kernel (v6.2.10). vainfo works fine, but decoding h264 with ffmpeg doesn't seem to work:

$ LIBVA_DRIVERS_PATH=~/git/libva-v4l2/build/src LIBVA_DRIVER_NAME=v4l2 ffmpeg -v debug -hwaccel vaapi -i ~/Big_Buck_Bunny_1080_10s_30MB.mp4 -f null -

......

[vost#0:0/wrapped_avframe @ 0xaaaae4252b00] Created video stream from input stream 0:0
[AVHWDeviceContext @ 0xaaaae43a0ec0] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: User environment variable requested driver 'v4l2'
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Trying to open /home/strongtz/git/libva-v4l2/build/src/v4l2_drv_video.so
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0xaaaae43a0ec0] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0xaaaae43a0ec0] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0xaaaae43a0ec0] VAAPI driver: v4l2.
[AVHWDeviceContext @ 0xaaaae43a0ec0] Driver not found in known nonstandard list, using standard behaviour.

......

[h264 @ 0xaaaae424ddc0] Format vaapi chosen by get_format().
[h264 @ 0xaaaae424ddc0] Format vaapi requires hwaccel h264_vaapi initialisation.
[h264 @ 0xaaaae424ddc0] Considering format 0x3231564e -> nv12.
[h264 @ 0xaaaae424ddc0] Picked nv12 (0x3231564e) as best match for yuv420p.
[AVHWFramesContext @ 0xffff70172d10] Created surface 0.
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: deriveImage 2
[AVHWFramesContext @ 0xffff70172d10] Direct mapping disabled: deriving image does not work: 1 (operation failed).
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Failed to create context: No surfaces to be created
[h264 @ 0xaaaae424ddc0] Failed to create decode context: 1 (operation failed).
[h264 @ 0xaaaae424ddc0] Failed to destroy decode context 0: 5 (invalid VAContextID).
[h264 @ 0xaaaae424ddc0] Failed setup for format vaapi: hwaccel initialisation returned error.
[h264 @ 0xaaaae424ddc0] Format vaapi not usable, retrying get_format() without it.
[h264 @ 0xaaaae424ddc0] Format yuv420p chosen by get_format().
[h264 @ 0xaaaae424ddc0] Reinit context to 1920x1088, pix_fmt: yuv420p

I added some debug prints, and the problem seems to be here:
https://github.com/mxsrc/libva-v4l2/blob/master/src/image.cc#L156
https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/hwcontext_vaapi.c#L672

[mxsrc]

Thanks for your input! I am trying to reproduce the error, assuming that my RK3399 is similar enough. So far I am getting a different (albeit still erroneous) behavior. Just to be sure, can you share the file you're trying to decode, so I am sure any difference isn't due to that?

[strongtz]

Sure. It's https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_30MB.mp4

[strongtz]

I'm using Arch Linux though, so the ffmpeg version is quite new. It's v7.1

[mxsrc]

I had a look at the issue and there appears to be a combination of issues, I'll details them here here for transparency mostly, but also development journaling on my side:

1. Currently, libva-v4l2 allocates buffers for the encoded frames at a fixed 1MiB size. This was sufficient for my testing material, but is insufficient for the P-Frames of the video you are decoding, and likely plenty of others. In general, V4L2 can dynamically determine the appropriate size for buffers, but specifically for MPEG2 decoding this unfortunately does not work, or did not work when I first tried to use it. I'm preparing a patchset that will use the dynamic buffer size by default and have an exception for MPEG2.
2. FFmpeg 7.1 changed how it manages its buffer pool. This decouples the libVA surfaces from the decode context which so far we've relied on. I'm figuring out how to solve this in a generic way.

[chewi]

Any news on this front? I just got the same with a different H.264 video using FFmpeg 8.0.1 with jc-kynesim's patches on a Quartz64 (RK3566).

[mxsrc]

I was not able to look into this last year due to a new day job. My new year's resolution is to make some progress on this, so I might have a fix in some time.

[22cms]

I've tried using this project on an Allwinner H313 with the Cedrus drivers running, and FFmpeg also seems to have a lot of trouble doing anything with it

[VAAPI @ 0xaaaaf7b0e730] libva: Failed to create context: No surfaces to be created
[mpeg2video @ 0xaaaaf7b256e0] Failed to create decode context: 1 (operation failed).
[mpeg2video @ 0xaaaaf7b256e0] Failed to destroy decode context 0: 5 (invalid VAContextID).
[mpeg2video @ 0xaaaaf7b256e0] Failed setup for format vaapi: hwaccel initialisation returned error.

I tried enabling VAAPI_TRACE but nothing that crazy seems to happen there either?

[55712.993930][ctx       none]=========vaQueryConfigProfiles ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994099][ctx       none]==========va_TraceCreateConfig
[55712.994104][ctx       none]	profile = 7, VAProfileH264High
[55712.994107][ctx       none]	entrypoint = 1, VAEntrypointVLD
[55712.994109][ctx       none]	num_attribs = 0
[55712.994247][ctx       none]=========vaCreateConfig ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994262][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994264][ctx       none]	config = 0x00000000
[55712.994275][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994279][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994281][ctx       none]	config = 0x00000000
[55712.994284][ctx       none]	attrib_list[0] =
[55712.994286][ctx       none]		type = 1
[55712.994288][ctx       none]		flags = 3
[55712.994290][ctx       none]		value.type = 1
[55712.994292][ctx       none]		value.value.i = 0x3231564e
[55712.994294][ctx       none]	attrib_list[1] =
[55712.994296][ctx       none]		type = 2
[55712.994297][ctx       none]		flags = 1
[55712.994299][ctx       none]		value.type = 1
[55712.994301][ctx       none]		value.value.i = 0x00000020
[55712.994303][ctx       none]	attrib_list[2] =
[55712.994304][ctx       none]		type = 3
[55712.994306][ctx       none]		flags = 1
[55712.994308][ctx       none]		value.type = 1
[55712.994310][ctx       none]		value.value.i = 0x00000800
[55712.994312][ctx       none]	attrib_list[3] =
[55712.994314][ctx       none]		type = 4
[55712.994315][ctx       none]		flags = 1
[55712.994317][ctx       none]		value.type = 1
[55712.994319][ctx       none]		value.value.i = 0x00000020
[55712.994321][ctx       none]	attrib_list[4] =
[55712.994323][ctx       none]		type = 5
[55712.994325][ctx       none]		flags = 1
[55712.994326][ctx       none]		value.type = 1
[55712.994328][ctx       none]		value.value.i = 0x00000800
[55712.994330][ctx       none]	attrib_list[5] =
[55712.994332][ctx       none]		type = 6
[55712.994334][ctx       none]		flags = 3
[55712.994336][ctx       none]		value.type = 1
[55712.994337][ctx       none]		value.value.i = 0x40000000
[55712.994344][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994353][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994355][ctx       none]	config = 0x00000000
[55712.994361][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994364][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994366][ctx       none]	config = 0x00000000
[55712.994368][ctx       none]	attrib_list[0] =
[55712.994370][ctx       none]		type = 1
[55712.994372][ctx       none]		flags = 3
[55712.994373][ctx       none]		value.type = 1
[55712.994375][ctx       none]		value.value.i = 0x3231564e
[55712.994377][ctx       none]	attrib_list[1] =
[55712.994379][ctx       none]		type = 2
[55712.994381][ctx       none]		flags = 1
[55712.994382][ctx       none]		value.type = 1
[55712.994384][ctx       none]		value.value.i = 0x00000020
[55712.994386][ctx       none]	attrib_list[2] =
[55712.994388][ctx       none]		type = 3
[55712.994389][ctx       none]		flags = 1
[55712.994391][ctx       none]		value.type = 1
[55712.994393][ctx       none]		value.value.i = 0x00000800
[55712.994395][ctx       none]	attrib_list[3] =
[55712.994396][ctx       none]		type = 4
[55712.994398][ctx       none]		flags = 1
[55712.994400][ctx       none]		value.type = 1
[55712.994402][ctx       none]		value.value.i = 0x00000020
[55712.994403][ctx       none]	attrib_list[4] =
[55712.994405][ctx       none]		type = 5
[55712.994407][ctx       none]		flags = 1
[55712.994409][ctx       none]		value.type = 1
[55712.994410][ctx       none]		value.value.i = 0x00000800
[55712.994412][ctx       none]	attrib_list[5] =
[55712.994414][ctx       none]		type = 6
[55712.994416][ctx       none]		flags = 3
[55712.994417][ctx       none]		value.type = 1
[55712.994419][ctx       none]		value.value.i = 0x40000000
[55712.994447][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994468][ctx       none]==========va_TraceDestroyConfig
[55712.994470][ctx       none]	config = 0x00000000
[55712.994479][ctx       none]=========vaDestroyConfig ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994525][ctx       none]==========va_TraceCreateSurfaces
[55712.994528][ctx       none]	width = 1920
[55712.994530][ctx       none]	height = 800
[55712.994532][ctx       none]	format = 1
[55712.994533][ctx       none]	num_surfaces = 1
[55712.994535][ctx       none]		surfaces[0] = 0x00000000
[55712.994538][ctx       none]	attrib_list[0] =
[55712.994540][ctx       none]		type = 1
[55712.994541][ctx       none]		flags = 2
[55712.994543][ctx       none]		value.type = 1
[55712.994545][ctx       none]		value.value.i = 0x3231564e
[55712.994547][ctx       none]	attrib_list[1] =
[55712.994548][ctx       none]		type = 6
[55712.994550][ctx       none]		flags = 2
[55712.994552][ctx       none]		value.type = 1
[55712.994554][ctx       none]		value.value.i = 0x00000001
[55712.994561][ctx       none]=========vaCreateSurfaces ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994576][ctx       none]=========vaDeriveImage ret = VA_STATUS_ERROR_OPERATION_FAILED, operation failed 
[55712.994638][ctx       none]=========vaQueryConfigProfiles ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994684][ctx       none]==========va_TraceCreateConfig
[55712.994687][ctx       none]	profile = 7, VAProfileH264High
[55712.994690][ctx       none]	entrypoint = 1, VAEntrypointVLD
[55712.994692][ctx       none]	num_attribs = 0
[55712.994700][ctx       none]=========vaCreateConfig ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994705][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994707][ctx       none]	config = 0x00000000
[55712.994713][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.994717][ctx       none]==========va_TraceQuerySurfaceAttributes
[55712.994719][ctx       none]	config = 0x00000000
[55712.994721][ctx       none]	attrib_list[0] =
[55712.994723][ctx       none]		type = 1
[55712.994724][ctx       none]		flags = 3
[55712.994726][ctx       none]		value.type = 1
[55712.994728][ctx       none]		value.value.i = 0x3231564e
[55712.994730][ctx       none]	attrib_list[1] =
[55712.994731][ctx       none]		type = 2
[55712.994733][ctx       none]		flags = 1
[55712.994735][ctx       none]		value.type = 1
[55712.994737][ctx       none]		value.value.i = 0x00000020
[55712.994739][ctx       none]	attrib_list[2] =
[55712.994740][ctx       none]		type = 3
[55712.994742][ctx       none]		flags = 1
[55712.994744][ctx       none]		value.type = 1
[55712.994745][ctx       none]		value.value.i = 0x00000800
[55712.994747][ctx       none]	attrib_list[3] =
[55712.994749][ctx       none]		type = 4
[55712.994751][ctx       none]		flags = 1
[55712.994753][ctx       none]		value.type = 1
[55712.994754][ctx       none]		value.value.i = 0x00000020
[55712.994756][ctx       none]	attrib_list[4] =
[55712.994758][ctx       none]		type = 5
[55712.994760][ctx       none]		flags = 1
[55712.994761][ctx       none]		value.type = 1
[55712.994763][ctx       none]		value.value.i = 0x00000800
[55712.994765][ctx       none]	attrib_list[5] =
[55712.994767][ctx       none]		type = 6
[55712.994768][ctx       none]		flags = 3
[55712.994770][ctx       none]		value.type = 1
[55712.994772][ctx       none]		value.value.i = 0x40000000
[55712.994778][ctx       none]=========vaQuerySurfaceAttributes ret = VA_STATUS_SUCCESS, success (no error) 
[55712.995301][ctx 0x00000000]==========va_TraceCreateContext
[55712.995308][ctx 0x00000000]	context = 0x00000000 va_trace_flag 0x1
[55712.995312][ctx 0x00000000]	profile = 7,VAProfileH264High entrypoint = 1,VAEntrypointVLD
[55712.995315][ctx 0x00000000]	config = 0x00000000
[55712.995318][ctx 0x00000000]	width = 1920
[55712.995320][ctx 0x00000000]	height = 800
[55712.995322][ctx 0x00000000]	flag = 0x00000001
[55712.995324][ctx 0x00000000]	num_render_targets = 0
[55712.995332][ctx       none]=========vaCreateContext ret = VA_STATUS_ERROR_OPERATION_FAILED, operation failed 
[55712.995467][ctx       none]=========vaDestroyContext ret = VA_STATUS_ERROR_INVALID_CONTEXT, invalid VAContextID 
[55712.995566][ctx       none]==========va_TraceDestroyConfig
[55712.995569][ctx       none]	config = 0x00000000
[55712.995696][ctx       none]=========vaDestroyConfig ret = VA_STATUS_SUCCESS, success (no error) 
[55712.995821][ctx       none]==========va_TraceDestroySurfaces
[55712.995825][ctx       none]		surfaces[0] = 0x00000000
[55712.995880][ctx       none]=========vaDestroySurfaces ret = VA_STATUS_SUCCESS, success (no error) 

Curiously, I tried using this fork of FFmpeg with V4L2_request support baked in, and that one works flawlessly, so it's not necessarily a problem with cedrus I guess?

[LesikEdelweiss]

I was not able to look into this last year due to a new day job. My new year's resolution is to make some progress on this, so I might have a fix in some time.

Hi!! Really nice work, I surely hope it works one day, as it seems to be the only way to get hardware decoders to work :(

Thank you for your work!