Skip to content

Conversation

dvrogozh
Copy link
Contributor

@dvrogozh dvrogozh commented Aug 26, 2025

Changes:

  1. Added DeviceInterface::initializeFiltersContext() API which returns device specific FilterGraph initialization settings
  2. Enabled filter graphs on SingleStreamDecoder level
  3. Switched CPU device interface to use above changes
  4. Enabled filter graphs in CUDA device interface to handle non-NV12 decoders output (10/12-bits videos) thru scale_cuda
    • Filter graph converts 10/12-bit videos to NV12 (as scale_cuda does not currently support conversion of YUV to RGB) and then cuda device interface converts NV12 to RGB (via existing NPP path)

Basically idea behind this change is the following. Let device interface to perform trivial and performance optimized conversions in the convertAVFrameToFrameOutput() method. If device interface can not handle conversion in the convertAVFrameToFrameOutput(), it can setup ffmpeg filters pipeline by returning valid description in initializeFiltersContext().

I tested the pipeline on 10-bit videos, h264 and h265. However, this setup should be valid for 12-bit videos which I did not try.

Note:

  1. On ffmpeg-n4.4 filters pipeline walls back to CPU as scale_cuda does not support format conversion in this ffmpeg version (was added from n5.0)
  2. More work might be needed to align scale_cuda converted outputs with CPU implementation. I do see the difference in the outputs likely due to different handling of color standards. This is something I was not able to overcome at the moment.

CC: @scotts @NicolasHug @eromomon

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 26, 2025
@dvrogozh
Copy link
Contributor Author

I forgot to mention that I did not change cpu device interface yet in a similar way to cuda (initializeFiltersContext() not implemented for cpu). This would be a next step if this design will be accepted.

@dvrogozh
Copy link
Contributor Author

I've added one more commit to update CPU device interface to use filters path introduce in SingleStreamDecoder and DeviceInterface. Still a draft as #831 needs to be reviewed first.

@dvrogozh dvrogozh marked this pull request as ready for review September 2, 2025 19:18
@dvrogozh
Copy link
Contributor Author

dvrogozh commented Sep 2, 2025

I've rebase the PR on top of the recently landed #831 prerequisite. Change is ready for review. @scotts @NicolasHug

@NicolasHug NicolasHug self-requested a review September 3, 2025 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant