Method to directly render the VideoDecoder to a Canvas element

[koush]

Currently, sending data from the VideoDecoder to Canvas involves:

• VideoDecoder.decode
• VideoFrame.createImageBitmap
• createImageBitmap again to flipY the imageOrientation (it seems that Chrome currently doesn't support the ImageBitmapOptions parameter on VideoFrame)
• CanvasContext.transferImageFromBitmap

Under the hood, this I suspect this converts VIdeoFrame YUV to ImageBitmap RGB, rather than directly rendering the YUV planes of the video frame in the webgl canvas. (Please let me know if I'm wrong)
There's also some lifecycle management on the frames and bitmaps to close and destroy them.

Ideally, the video frames (or perhaps decoder) should have a method to directly render to a canvas. This is similar to how Android's decoder operates, and for lack of better description "feels" like it is closer to achieving zero copy.

[sandersdan]

Edit: ImageBitmapOptions is in the spec and in Chrome's implementation. If it doesn't work in Chrome Canary please file a Chromium bug.

In Chrome's implementation, createImageBitmap does indeed do a conversion to RGB. There are optimizations we still plan to make, but it's not yet certain what faction of cases will continue to require a conversion.

There isn't currently a way to bind WebGL textures to images, but this is something that operating systems can usually do. I've heard that WebXR is investigating whether there is a path forward there.

I'm hopeful that WebGPU will be more amenable to this sort of usage. There has been some discussion but no agreed plan, see gpuweb/gpuweb#625.

[koush]

Thanks for the implementation details, that is good to know. I look forward to any future optimizations.

Looking into VideoFrame interface, it does seem like the YUV Planes are accessible for reading (though it presumably requires a memcpy). I can use that to render the YUV onto a webgl canvas context. This will likely be more efficient than performing a userspace YUV->RGB->flipY->render.

https://wicg.github.io/web-codecs/#plane

If there's a better method to render h264 nal units directly to a canvas than what I described above, please let me know!

[koush]

I made the changes to draw the VideoFrame yuv planes via webgl and can confirm that the rendering is significantly faster. Latency and CPU usage both decreased dramatically.

Setting up and rendering via a GL context is more involved, so convenience methods to do this would be great. Right now, the createImageBitmap/transferImageFromBitmap is deceptively expensive and slow. Problematically, it is seemingly the fastest/direct way to draw to the screen/canvas.

[koush]

Update:
I ran into a few issues with decoding and then rendering to a canvas.

On ChromeOS, and other hardware accelerated platforms, VideoFrame.planes may be null. Furthermore, VIewFrame.createImageBitmap is returning blank/black frames.

This means on some (indeterminable at runtime) platforms, the decoder is completely non-functional, as the frames can't be retrieved in any fashion.

Chrome 86.

[jchen10]

For ChromeOS issue, there is a bug at https://bugs.chromium.org/p/chromium/issues/detail?id=1137947
Could you upload your error log in chrome://gpu there? thanks!

[chcunningham]

+1 to jchen's suggestion.

For folks reading along: the case of null planes for hw-decoded frames is a known limitation of chrome's current impl, but the bug in createImageBitmap() breaks the intended workaround. In addition to getting the bug fixed, we're planning to spec an API for format conversion (e.g. null -> i420).

[koush]

@chcunningham Unfortunately, even the createImageBitmap is an unusable workaround for my use case, as it does an unaccelerated YUV to RGB conversion and then memcpy that onto the canvas. There is significant latency and freezing on low power devices. It's actually faster to use a WebAssembly h264 decoder, because the YUV planes are accessible and can be rendered directly with WebGL.

The ideal solution as mentioned in title is just a way to render directly to a canvas from a decoder. The conversions, while nice to have, are a roundabout way to get the decoder rendering to an element, which I imagine will be the primary use case.

[koush]

will get a chrome://gpu for the bug shortly.

[chcunningham]

The ideal solution as mentioned in title is just a way to render directly to a canvas from a decoder. The conversions, while nice to have, are a roundabout way to get the decoder rendering to an element, which I imagine will be the primary use case.

Agreed. We're tracking these optimizations here
https://bugs.chromium.org/p/chromium/issues/detail?id=1141182

[AlexVestin]

Agreed. We're tracking these optimizations here

Can this also apply to have a zero-copy from a canvas to a VideoEncoder, in cases where hardware acceleration is available?

[chcunningham]

@koush and others, we have some updates!

• we've just landed support for painting VideoFrames via drawImage and texImage2d. This will release in canary channel shortly (check here: https://chromiumdash.appspot.com/commit/9229440513bd0995595a6e3ab72686d3f7f1c8d6)
• we've also done some work to expose extra pixel formats (e.g. NV12) and map gpu backed memory for copy access. I expect you really just want drawImage^, but just a FYI that your chromeos decodes may now give you pixel data depending on your stream's underlying pixel format

Please let us know how it works!

Can this also apply to have a zero-copy from a canvas to a VideoEncoder, in cases where hardware acceleration is available?

@AlexVestin, most encoders will require that we at least do an RGB -> YUV conversion before encoding. This conversion can happen on the GPU, so its not awful. Still, there are some additional copies in the path of Canvas -> ImageBitmap -> Encoder which could be better optimized. Can you share more on your use case? I may be able to suggest a more efficient path.

[koush]

• we've just landed support for painting VideoFrames via drawImage and texImage2d. This will release in canary channel shortly (check here: https://chromiumdash.appspot.com/commit/9229440513bd0995595a6e3ab72686d3f7f1c8d6)

It looks like you can feed VideoFrame directly to texImage2d! This is great!
https://chromium.googlesource.com/chromium/src.git/+/9229440513bd0995595a6e3ab72686d3f7f1c8d6%5E%21/#F9

let frame = new VideoFrame('ABGR', [argbPlane], vfInit);
gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, 0, gl.RGBA, gl.UNSIGNED_BYTE, frame);

[BenV]

@chcunningham I implemented this in a test app this morning, and unfortunately I am seeing quite a bit of overhead when using VideoFrame as opposed to an HTMLVideoElement with texImage2d to process frames from my webcam on both PC and Mac. It takes roughly twice as long when using the VideoFrame method. Is this expected?

FWIW I've done quite a bit of benchmarking on this stuff while trying to find the most optimal way to manipulate a camera feed, and these numbers look pretty close to what I was seeing for the total time when using window.createImageBitmap on the HTMLVideoElement and then passing the resulting ImageBitmap into texImage2d.

[koush]

Have implemented the new VideoFrame canvas rendering and can confirm that this is working with hardware acceleration on my Mac now. Prior to this Canary build, the previous method had no way to render the frame, as createImageBitmap was returning a blank bitmap.

[AlexVestin]

@chcunningham
The createImageBitmap call with a canvas averaged at around 10ms, so I assumed it was copying the frame back from the GPU, but I might have been mistaken about that.

Can you share more on your use case? I may be able to suggest a more efficient path.

The use case is encoding frame by frame from a canvas for animation rendering, in cases where the user is not be able to encode it in real time