Display support opencv headless #94961

hakehuang · 2025-08-25T13:30:30Z

for test machine without GUI, we need support open-cv-headless mode.
Enable mimxrt1170_evk@A
parse environment variable, which is missed in former PR.

hakehuang · 2025-08-25T13:31:08Z

test log

DEBUG   - run status: mimxrt1170_evk@A/mimxrt1176/cm7/zephyr/tests.drivers.display.check.shield passed
INFO    - 1/1 mimxrt1170_evk@A/mimxrt1176/cm7 tests.drivers.display.check.shield           PASSED (device: JTAQCQAR, 72.943s <zephyr>)
INFO    -                                    tests.drivers.display.check.shield.test_display_harness                     PASSED

INFO    - 4 test scenarios (4 configurations) selected, 3 configurations filtered (3 by static filter, 0 at runtime).
Summary
├── Total test suites: 4
├── Processed test suites: 4
│   ├── Filtered test suites: 3
│   │   ├── Filtered test suites (static): 3
│   │   └── Filtered test suites (at runtime): 0
│   └── Selected test suites: 1
│       ├── Skipped test suites: 0
│       ├── Passed test suites: 1
│       ├── Built only test suites: 0
│       ├── Failed test suites: 0
│       └── Errors in test suites: 0
└── Total test cases: 4
    ├── Filtered test cases: 3
    └── Selected test cases: 1
        ├── Passed test cases: 1
        ├── Skipped test cases: 0
        ├── Built only test cases: 0
        ├── Blocked test cases: 0
        ├── Failed test cases: 0
        └── Errors in test cases: 0
INFO    - 1 of 1 executed test configurations passed (100.00%), 0 built (not run), 0 failed, 0 errored, with no warnings in 102.90 seconds.
INFO    - 1 of 1 executed test cases passed (100.00%) on 1 out of total 1152 platforms (0.09%).
INFO    - 1 test configurations executed on platforms, 0 test configurations were only built.

hakehuang · 2025-08-29T04:15:43Z

@JarmouniA @danieldegrasse @jfischer-no @nashif Please help to review, this enhance the test for headless(No Graphic) condition, also fix a missing in test path parser. Thanks

JarmouniA · 2025-08-29T08:08:54Z

Cc @Finomnis

Finomnis · 2025-08-29T08:45:30Z

Cc @Finomnis

Not sure if I can help, didn't work with this part of Zephyr yet

JarmouniA · 2025-08-29T15:26:55Z

Cc @Finomnis

Not sure if I can help, didn't work with this part of Zephyr yet

Sorry, I thought the PR was about headless SDL for some unknown reason, hence the ping :)

jfischer-no · 2025-09-01T10:09:41Z

@JarmouniA @danieldegrasse @jfischer-no @nashif Please help to review, this enhance the test for headless(No Graphic) condition, also fix a missing in test path parser. Thanks

 tests: display: support headless mode

1. support headless mode
2. add parser for environment path

@hakehuang I think the commit message could be improved. I suggest writing about the motivation and why, rather than listing what has actually changed in bullet form.

hakehuang · 2025-09-01T13:08:12Z

@hakehuang I think the commit message could be improved. I suggest writing about the motivation and why, rather than listing what has actually changed in bullet form.

@jfischer-no I update the commit message, but not sure is it clear enough.

for test machine without display, need support open-cv-headless mode.

hakehuang · 2025-09-03T03:33:01Z

@JarmouniA @danieldegrasse @jfischer-no @nashif Please help to review

for test machine without display, need support open-cv-headless mode. 1. support headless mode 2. add parser for environment path Signed-off-by: Hake Huang <[email protected]>

enable mimxrt1170_evk@A add timeout, as for big screen, the fist step takes a long time Signed-off-by: Hake Huang <[email protected]>

hakehuang · 2025-09-10T03:21:50Z

@josuah please help to take a look, thanks.

josuah · 2025-09-10T10:11:09Z

Hello! I tried again in a debootstrap environment where display is not available ($DISPLAY is empty) and it seems like the Qt libraries did complain.

This is a known issue when trying to load the Qt libraries when X11 does not work.

(zephyrproject) ~/zephyrproject/zephyr/scripts/pylib/display-twister-harness HEAD $ python -m camera_shield.main --config config.yaml
Authorization required, but no authorization protocol specified

qt.qpa.xcb: could not connect to display :0
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/panoramix/zephyrproject/.venv/lib/python3.13/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted
(zephyrproject) ~/zephyrproject/zephyr/scripts/pylib/display-twister-harness HEAD $ echo "$DISPLAY"

(zephyrproject) ~/zephyrproject/zephyr/scripts/pylib/display-twister-harness HEAD $

However, with this small patch, it did not have any issue:

diff --git a/scripts/pylib/display-twister-harness/camera_shield/main.py b/scripts/pylib/display-twister-harness/camera_shield/main.py
index b0afd6ec2c6..6bf96ccb4a8 100644
--- a/scripts/pylib/display-twister-harness/camera_shield/main.py
+++ b/scripts/pylib/display-twister-harness/camera_shield/main.py
@@ -94,8 +94,9 @@ class Application:
 
                 # Maintain OpenCV event loop
                 try:
-                    if cv2.waitKey(1) == 27:  # ESC key
-                        break
+                    break
+                    #if cv2.waitKey(1) == 27:  # ESC key
+                    #    break
                 except Exception as e:
                     error_msg = str(e).lower()
                     if any(

And I could get these results:

(zephyrproject) ~/zephyrproject/zephyr/scripts/pylib/display-twister-harness HEAD $ python -m camera_shield.main --config config.yaml
Camera 0 Settings:
Resolution: 1280                x720
  FPS: 10.00
  Auto Focus: ON
  Auto White Balance: ON
initialize
compare
/home/panoramix/zephyrproject/zephyr/scripts/pylib/display-twister-harness/./fingerprints not exist
VideoSignaturePlugin result: no match

The problem is that Qt calls abort(), which is always a problem as we cannot recover from this error as the Qt library exits before returning to Python.

hakehuang · 2025-09-10T10:21:04Z

The problem is that Qt calls abort(), which is always a problem as we cannot recover from this error as the Qt library exits before returning to Python.

@josuah, skip calling the GUI related function can avoid such issue, but the recommend way is to replace is to do below

if you server does not have display please do below

pip uninstall opencv-python
pip install opencv-python-headless
export QT_QPA_PLATFORM=offscreen

hakehuang · 2025-09-10T10:21:34Z

The problem is that Qt calls abort(), which is always a problem as we cannot recover from this error as the Qt library exits before returning to Python.

@josuah, skip calling the GUI related function can avoid such issue, but the recommend way is to replace is to do below

if you server does not have display please do below

pip uninstall opencv-python
pip install opencv-python-headless
export QT_QPA_PLATFORM=offscreen

josuah · 2025-09-10T10:22:57Z

How about adding a flag to control whether a GUI is present or not?

For instance:

if --headless is selected, then show_frame() calls cv2.imwrite() instead of cv2.imshow(), and prints frame captured at ./path_to_file.png and calls input() instead of cv2.waitKey().
if --headless is absent, this runs the same script as now.

Or do you prefer to always to auto-detection of the GUI?
If so the current PR improves the support for headless mode, no problem.

Thank you for the improvement!

josuah

Thank you, I must have missed the pip install opencv-python-headless line. Sorry about that. It all works on first try now!

JarmouniA · 2025-09-10T11:24:54Z

pip install opencv-python-headless

Should be added in the doc.

hakehuang · 2025-09-11T02:57:05Z

pip install opencv-python-headless

Should be added in the doc.

sure, update the document 109cc06 @JarmouniA , @josuah

scripts/pylib/display-twister-harness/camera_shield/README.rst

josuah · 2025-09-11T10:45:00Z

The documentation is not being built, so CI does not trigger any warning.

You may see that it is mixing Markdown and RST syntax, and I guess us all developers are too used to Markdown to realize it is invalid syntax in RST:

https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/pylib/display-twister-harness/camera_shield/README.rst

Maybe it can be improved in a different PR as this is not related to documentation.

Thanks!

hakehuang · 2025-09-11T12:13:21Z

Maybe it can be improved in a different PR as this is not related to documentation.

Thanks!

@JarmouniA. I update the document and check with rstcheck now the format should be right. Thanks to point this out.

add Installation Guide section Signed-off-by: Hake Huang <[email protected]>

sonarqubecloud · 2025-09-12T03:27:20Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kartben · 2025-10-20T13:25:01Z

scripts/pylib/display-twister-harness/camera_shield/README.rst

+   - clone code
+
+     ::
+
+        git clone https://github.com/hakehuang/camera_shield


this looks wrong? also, this sould be included in the Zephyr documentation proper, not as a separate README that is virtually invisible to anyone browsing the online docs. Thanks!

@kartben , I move the whole https://github.com/hakehuang/camera_shield into the zephyr already, should I remove this file?

@kartben , I move the whole hakehuang/camera_shield into the zephyr already, should I remove this file?

not remove it, we want proper docs for this :)
see #98398

zephyrbot added area: Twister Twister area: Display labels Aug 25, 2025

zephyrbot requested review from JarmouniA, KamilxPaszkiet, LukaszMrugala, PerMac, VynDragon, danieldegrasse, gchwier, golowanow, jfischer-no and nashif August 25, 2025 13:31

zephyrbot assigned nashif Aug 25, 2025

hakehuang force-pushed the display_support_opencv_headless branch 6 times, most recently from 96a9f34 to 68c8642 Compare August 26, 2025 05:23

hakehuang force-pushed the display_support_opencv_headless branch from 68c8642 to ae888ba Compare September 1, 2025 13:07

hakehuang force-pushed the display_support_opencv_headless branch 2 times, most recently from 9f5d197 to 804953a Compare September 2, 2025 08:14

hakehuang added 2 commits September 5, 2025 16:07

tests: display: support headless mode

865ad98

for test machine without display, need support open-cv-headless mode. 1. support headless mode 2. add parser for environment path Signed-off-by: Hake Huang <[email protected]>

tests: display: display_check: add test time and enable mimxrt1170

a5409fa

enable mimxrt1170_evk@A add timeout, as for big screen, the fist step takes a long time Signed-off-by: Hake Huang <[email protected]>

hakehuang force-pushed the display_support_opencv_headless branch from a00ea65 to a5409fa Compare September 5, 2025 08:08

JarmouniA previously approved these changes Sep 5, 2025

View reviewed changes

josuah previously approved these changes Sep 10, 2025

View reviewed changes

hakehuang dismissed stale reviews from josuah and JarmouniA via 109cc06 September 11, 2025 03:10

hakehuang requested review from JarmouniA and josuah September 11, 2025 03:17

JarmouniA reviewed Sep 11, 2025

View reviewed changes

scripts/pylib/display-twister-harness/camera_shield/README.rst Outdated Show resolved Hide resolved

scripts/pylib/display-twister-harness/camera_shield/README.rst Outdated Show resolved Hide resolved

hakehuang force-pushed the display_support_opencv_headless branch 2 times, most recently from 22488fe to 86b5956 Compare September 11, 2025 12:11

doc: scripts: display_harness: update document on installation

bb22e13

add Installation Guide section Signed-off-by: Hake Huang <[email protected]>

hakehuang force-pushed the display_support_opencv_headless branch from 86b5956 to bb22e13 Compare September 12, 2025 03:00

hakehuang requested a review from JarmouniA September 12, 2025 05:00

nashif approved these changes Sep 13, 2025

View reviewed changes

JarmouniA approved these changes Sep 13, 2025

View reviewed changes

cfriedt merged commit 504fb01 into zephyrproject-rtos:main Sep 13, 2025
47 checks passed

hakehuang mentioned this pull request Sep 15, 2025

samples: lvgl: enhacements for CI check updates for NXP platforms #84458

Draft

kartben reviewed Oct 20, 2025

View reviewed changes

Display support opencv headless #94961

Display support opencv headless #94961

Uh oh!

Conversation

hakehuang commented Aug 25, 2025

Uh oh!

hakehuang commented Aug 25, 2025

Uh oh!

hakehuang commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JarmouniA commented Aug 29, 2025

Uh oh!

Finomnis commented Aug 29, 2025

Uh oh!

JarmouniA commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfischer-no commented Sep 1, 2025

Uh oh!

hakehuang commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hakehuang commented Sep 3, 2025

Uh oh!

hakehuang commented Sep 10, 2025

Uh oh!

josuah commented Sep 10, 2025

Uh oh!

hakehuang commented Sep 10, 2025

Uh oh!

hakehuang commented Sep 10, 2025

Uh oh!

josuah commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah left a comment

Choose a reason for hiding this comment

Uh oh!

JarmouniA commented Sep 10, 2025

Uh oh!

hakehuang commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josuah commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hakehuang commented Sep 11, 2025

Uh oh!

sonarqubecloud bot commented Sep 12, 2025

Quality Gate passed

Uh oh!

Uh oh!

kartben Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

hakehuang Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

kartben Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

hakehuang commented Aug 29, 2025 •

edited

Loading

JarmouniA commented Aug 29, 2025 •

edited

Loading

hakehuang commented Sep 1, 2025 •

edited

Loading

josuah commented Sep 10, 2025 •

edited

Loading

hakehuang commented Sep 11, 2025 •

edited

Loading

josuah commented Sep 11, 2025 •

edited

Loading