video: H.264 video format support #95407

thecapn32 · 2025-09-03T13:02:02Z

This is an attempt to support H.264 video compressed format on native_sim.
There is H.264 video provided in C array .h file and what is doing now is read video data from that file and send it over TCP. Due to this the frame rates are wrong.

This works needs to be done:

add mechanism to send h264 frames to maintain 30FPS
dynamic way to adjust video format in video-sw-generator ( now the H.264 format is static)

…ompressed formats

github-actions · 2025-09-03T13:02:39Z

Hello @thecapn32, and thank you very much for your first pull request to the Zephyr project!
Our Continuous Integration pipeline will execute a series of checks on your Pull Request commit messages and code, and you are expected to address any failures by updating the PR. Please take a look at our commit message guidelines to find out how to format your commit messages, and at our contribution workflow to understand how to update your Pull Request. If you haven't already, please make sure to review the project's Contributor Expectations and update (by amending and force-pushing the commits) your pull request if necessary.
If you are stuck or need help please join us on Discord and ask your question there. Additionally, you can escalate the review when applicable. 😊

josuah

Thanks for this improvement! Timely with the #92884 pull request doing the same thing but with real hardware, as well as every future video device.

Some quick feedback first, I will have to give a more in-depth round later.

josuah · 2025-09-03T13:12:30Z

drivers/video/video_sw_generator.c

+		.fmt.width = 640,                                                                  \
+		.fmt.height = 320,                                                                 \
+		.fmt.pitch = 0,                                                                    \
+		.fmt.pixelformat = VIDEO_PIX_FMT_H264,                                             \


This was probably convenient for testing, but it is better to keep it to what it was, and instead use the video_set_format() API from the application to select H.264.

josuah · 2025-09-03T13:14:13Z

samples/drivers/video/tcpserversink/src/main.c

+    if (video_bits_per_pixel(fmt.pixelformat) > 0) {
+        buffer_size = fmt.pitch * fmt.height;
+    } else {
+        buffer_size = fmt.width * fmt.height / 10;


This is probably going to need some other strategy as hard to predict the compression ratio.

This might as well be something the user decides, depending on what is expected. For instance, buffer_size = fmt.width * fmt.height / CONFIG_VIDEO_MIN_COMPRESSION_RATIO, or even buffer_size = CONFIG_VIDEO_COMPRESSED_BUFFER_SIZE.

josuah · 2025-09-03T13:17:15Z

include/zephyr/drivers/video.h

+/**
+ * H264 without start code
+ */
+#define VIDEO_PIX_FMT_H264_MVC VIDEO_FOURCC('M', '2', '6', '4')
+


Maybe we can keep 3D TV H.264 for another day and not introduce H264_MVC for now? https://en.wikipedia.org/wiki/Multiview_Video_Coding

In order to move forward on PR #92884, I've cherry-picked this commit but without the M264 definition.

drivers/video/video_sw_generator.c

josuah · 2025-09-03T13:26:07Z

drivers/video/video_h264_test.h

Clicking view this file shows something like this:

unsigned char h264_test_data[] = { 0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0xc0, 0x1e, 0x8c, 0x68, 0x0a, 0x03, 0xdb, 0x01, 0x01, 0xe1, 0x10, 0x8d, 0x40, 0x00, 0x00, 0x00, 0x01, 0x68, 0xce, 0x3c, 0x80, 0x00, 0x00, 0x00, 0x01, 0x65, 0xb8, 0x00, 0x04, 0x08, 0xf8, 0x84, 0x44, 0x44, 0x18, 0x11, 0xd5, 0x93, 0x80, 0xba, 0x90, 0x00, 0x75, 0x5f, 0xe0, 0x01, 0xcb, 0x33, 0x44, 0x51, 0xa0, 0xd4, 0xdd, 0xfe, ... 0xb4, 0x32, 0x5b, 0x5f, 0xf3, 0xc6, 0x2d, 0xc6, 0x1e, 0xb1, 0xfe, 0x87, 0xbc, 0x0d, 0xdb, 0x8a, 0x58, 0x4b, 0xf9, 0xe0, 0x9d, 0x44, 0xe7, 0xaf, 0x08, 0x5b, 0x9e, 0xa0 }; unsigned int h264_test_data_len = 315304;

Maybe it is interesting to include the file as a binary. I think there are some places in Zephyr where this is done and will try to give pointers on how to do this.

Related #42580 (comment)

I think the best approach would be to remove the blob, and add instructions in the sample doc showing how to generate the header from a video file.

This would also remove the amount of data on the repo.
Gstreamer and FFmpeg can generate files.

@JarmouniA Would it make sense to add these commands to CMake so that the user does not have to manually copy-paste a command in order to build? This would prevent to use it for i.e. CI.

It does not hurt to also document it though.

Be it in the generated .h file, or in the hard-coded .h file, this needs to be declared as const, so that it fits in the boards it is tested on. Currently we run out of RAM:

/opt/toolchains/zephyr-sdk-0.17.2/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: zephyr/zephyr_pre0.elf section `datas' will not fit in region `RAM'

Could you paste the ffmpeg command you used if you remember it?
I can try adding a bit of CMake here to generate the headers when H.264 is enabled and paste it there.

yes!
To generate the H.264 file

ffmpeg -f lavfi -i testsrc2=duration=5:size=640x480:rate=30 \ -c:v h264 -b:v 500k -an test.h264

you can play it via:

ffplay test.h264

Convert it to C array:

xxd -i test.h264 > video_h264_test.h

For later: an example of how a custom command might be integrated in CMakeLists.txt:

zephyr/samples/subsys/llext/edk/ext1/CMakeLists.txt

Lines 24 to 33 in 0520dfe

add_custom_command(

OUTPUT

${PROJECT_BINARY_DIR}/${PROJECT_NAME}.llext

${PROJECT_BINARY_DIR}/${PROJECT_NAME}.inc

COMMAND ${CMAKE_C_COMPILER} ${COMPILE_OPTIONS_PROP}

-o ${PROJECT_BINARY_DIR}/${PROJECT_NAME}.llext

${PROJECT_SOURCE_DIR}/src/main.c

COMMAND xxd -ip ${PROJECT_NAME}.llext

${PROJECT_NAME}.inc

)

It seems like xxd is currently accepted as a dependency in Zephyr already, we can use the same approach.

sonarqubecloud · 2025-09-03T14:35:47Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

josuah

Sorry for the long wait!

I think once the big .h is turned into a small line of ffmpeg this can be merged as a first early version that can be improved later.

josuah · 2025-10-07T11:10:28Z

drivers/video/video_h264_test.h

Be it in the generated .h file, or in the hard-coded .h file, this needs to be declared as const, so that it fits in the boards it is tested on. Currently we run out of RAM:

/opt/toolchains/zephyr-sdk-0.17.2/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: zephyr/zephyr_pre0.elf section `datas' will not fit in region `RAM'

drivers/video/video_sw_generator.c

josuah · 2025-10-07T11:21:11Z

drivers/video/video_h264_test.h

Could you paste the ffmpeg command you used if you remember it?
I can try adding a bit of CMake here to generate the headers when H.264 is enabled and paste it there.

thecapn32 added 4 commits September 3, 2025 16:10

include: driver: video: add h264 pixel format support

d50dd0c

driver: video: add h264 compressed data in array format

fa5a00f

driver: video: sw_generator: support h264 pixel format

366a6bd

samples: drivers: video: tcpserversink: fix buffer allocation for unc…

ef080c2

…ompressed formats

josuah self-requested a review September 3, 2025 13:09

thecapn32 force-pushed the video-sw-generator branch from dcda85f to ef080c2 Compare September 3, 2025 13:25

josuah reviewed Sep 3, 2025

View reviewed changes

thecapn32 requested a review from josuah September 3, 2025 13:30

josuah mentioned this pull request Sep 11, 2025

video: add video compression support to tcpserversink sample #95862

Merged

hfruchet-st mentioned this pull request Oct 1, 2025

video: introduction of STM32 VENC driver for H264 hardware video compression #92884

Merged

josuah reviewed Oct 7, 2025

View reviewed changes

	add_custom_command(
	OUTPUT
	${PROJECT_BINARY_DIR}/${PROJECT_NAME}.llext
	${PROJECT_BINARY_DIR}/${PROJECT_NAME}.inc
	COMMAND ${CMAKE_C_COMPILER} ${COMPILE_OPTIONS_PROP}
	-o ${PROJECT_BINARY_DIR}/${PROJECT_NAME}.llext
	${PROJECT_SOURCE_DIR}/src/main.c
	COMMAND xxd -ip ${PROJECT_NAME}.llext
	${PROJECT_NAME}.inc
	)

Uh oh!

video: H.264 video format support #95407

Are you sure you want to change the base?

video: H.264 video format support #95407

Conversation

thecapn32 commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

josuah left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thecapn32 Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Sep 3, 2025

Quality Gate passed

Uh oh!

josuah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

josuah left a comment •

edited

Loading

josuah Sep 3, 2025 •

edited

Loading

thecapn32 Oct 7, 2025 •

edited

Loading

josuah Oct 7, 2025 •

edited

Loading