-
Notifications
You must be signed in to change notification settings - Fork 8.1k
VIDEO: GC2145/stm32_dcmi support for CROP/Snapshot #93797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
558b218
to
aa416e3
Compare
@josuah @mjs513 @dkalowsk @iabdalkader and all: As @josuah mentioned in my previous PR:
Which makes sense: But now wondering about a few details. In my own test case app I have:
And the CONF file has:
But if I was not using my updated fmts, which allows more resolutions, I would have done FRAME_WIDTH=640 and HEIGHT=480 So now assume:
With this, the call to video_set_format would have a width=640 and height= 480 Note: If you now (first commit) call video_get_selection with VIDEO_SEL_TGT_NATIVE_SIZE, it will return 800x600 So now to do step 2) to crop it to 480x320 - Currently I ignore the passed in top and left in the crop rectangle, but that So with this setup, I would expect on the TGT_CROP that I should be able to pass in rectangles in the range: But in this case: should step 1) have set the top=0, left=0 or should it instead of set it to 80,60? Thanks |
Having problems with this PR first commit on getting signoff to be accepted: I actually copy/pasted the signoff line from previous PRS which worked then but not on this one?
All of which failed like: -- Run compliance checks on patch series (PR): Identity.txt#L0See https://docs.zephyrproject.org/latest/contribute/guidelines.html#commit-guidelines for more details 52ea30be81e17759c1725a3ef7e7c2c64d6ed5c0: Signed-off-by line (Signed-off-by: Kurt Eckhardt ) does not follow the syntax: First Last .52ea30be81e17759c1725a3ef7e7c2c64d6ed5c0: Signed-off-by line (Signed-off-by: Kurt Eckhardt [email protected]) does not follow the syntax: First Last . |
f1d99dd
to
acb2521
Compare
More mumbling to self ;)
Reworking: Currently I have the crop code recalculate most of the window and crop registers. Will instead have it As the setting the crop updates the fmt structure:
Why? Because the buffer management code requires the buffers to be that size:
So for example if I setup for the GC2145 camera to output 480x320 the buffer size needed is: 307200 bytes This also effects the range of crop LEFT and TOP to fit the 480x320 within the range of 640x480... |
3b43aca
to
019f84b
Compare
Note: this all uses the Zephyr updates from my PR zephyrproject-rtos/zephyr#93797 Which I added to the STM dcmi driver the ability to have the camera work in snapshot mode instead of in continuous video mode. This allows for example that we start the camera it grabs a frame and stops, we then take the buffer and process it, and repeat this. This helps minimize how much the SDRAM gets used concurrently. In addition, I added to the VIDEO_STM32_DCMI and GC2145 the ability to use some of the new set_selection and get_selection code that was added for the DCMIPP. In particular the DCMI simply forwards these messages to the camera if it defined thise apis... And with this it allows you to setup a viewport into the frame. For example: You can setup the frame on the GC2145 to be 800x600 and then define a view port to be 480x320 to fill an ST7796/ILI9486 tft display or you could do it 400x240 to half fill the GIGA display. You can also move that view port around within the frame (pan) I have examples that do this on Portenta H7 on the ST7796 display and another one that does this on the GIGA display shield. Still WIP as we probably need to refine the APIS and the like
0cb0ea4
to
74b3afe
Compare
@avolmat-st I just updated the last commit from yesterday to fix the issue I was running into with my test app, that has a short timeout. It ran into issue where it had not completed getting a frame, but by the next time I called, it had completed it. Code did not handle As noted: the non-snapshot mode appears to be stable and recovering. Snapshot mode is not recovering: I believe I mentioned before that I found at times doing the start and maybe stop as part of the callback could be problematic. my WAG is that when you call the start with the snapshot mode option, the API is setup to wait until it finds the start of a new frame, which in cases could be more or less waiting a whole frame, and maybe doing that in Will try temporarily undoing that part of the changes and see if it improves again. |
bc0ae70
to
1884f61
Compare
@avolmat-st @josuah @JarmouniA - I pushed up some changes to the snapshot mode, that if it finds it already I believe I Should not hard code, but should allow the program to set it. Not sure which was is best to do this. Should I also change to init controls for setting being in snapshot? Thanks EDIT: Been running now for a few hours, >45K frames received and displayed on display.
Note: I think my sketch semi paused when my computer went to sleep, probably because it could not output to the USB
|
249a951
to
30f3420
Compare
@avolmat-st - I redid the timeouts code mentioned yesterday. I now do it at the end of the dequeue if it did Note: I split the dequeue up into two functions. The main one checks if it is snaphot and if so calls off to different function. I have run both ways this morning and both are recovering from errors now. The nice this is having the setting and checking I am thinking maybe time to squash back to just two commits. Thoughts? |
Error recovery code done in callback for both modes. Timeout fixes/changes in snapshot. Two different timeouts. First is a timeout for the call to dequeue. Which can return before a snapshot completes. Additional calls to dequeue will continue to wait if necessary, for a frame to be received. A second timeout was added, which can be configured as parameter to DCMI. This handles cases where the snapshot start completes successfully, but does not return a frame in the prescribed time frame. It currently defaults to one second. Signed-off-by: Kurt Eckhardt <[email protected]>
30f3420
to
a27e1a2
Compare
|
@avolmat-st @mjs513 and all. I believe that it is working reasonably well. Both continuous and snapshot. Today I ported it back In: stm32_dma_init
switched back to:
And in the newer stuff it has:
With these changes I am able to read in frames and display them on the GIGA display shield. Without these changes With them it has run for a long period of time. Currently running with 3 frame buffers (no snapshot) |
@KurtE About the dma changes. What is the dts dma configuraiton you're using ? |
Sorry, not sure what exactly you are asking, but will guess: With the Arduino build which is based off of zephyr v4.2.0 on the Arduino Giga board the DMA on the dcmi is configured:
Since v4.2.0 released, the DCMI DMA code (video_stm32_dcmi) has changed in a couple of PRs.
And your commit: drivers: stm32: Make use of new GET_INSTANCE DMA macro On Current stuff: I am using:
And pulled in the PR: Note: I tried on my testing of camera code back on ArduinoCore-zephyr tried to pull in the updated version (minus new macro) and neither the camera nor display worked properly. Not sure it interferred with the Display driver. Don't know how to test |
This was extracted from my snapshot PR: zephyrproject-rtos#93797 Been wondering if we should split off from there and at least get the simple recovery mechanism in place? Signed-off-by: Kurt Eckhardt <[email protected]>
This was extracted from my snapshot PR: zephyrproject-rtos#93797 Been wondering if we should split off from there and at least get the simple recovery mechanism in place? Signed-off-by: Kurt Eckhardt <[email protected]>
This was extracted from my snapshot PR: zephyrproject-rtos#93797 Been wondering if we should split off from there and at least get the simple recovery mechanism in place? Signed-off-by: Kurt Eckhardt <[email protected]>
@josuah @ngphibang @avolmat-st @mjs513 I am wondering if it would be best to maybe split this up into two (or three) Pull Requests a) b) Support for the cropping in GC2145 plus DCMI c) Snapshot mode - which I still think would be good for longer term, but it would be nice to have a) and b) soon - hopefully in time for next Arduino zephyr release as it at least Thanks |
Hi @KurtE, sorry for the long delay without a reply. Yeah, it would be great to have this merged before the freeze period start. I will look again at the PR very very shortly. |
Thanks, As I mentioned if there are issues with doing snapshot, that could it split it up... So this morning I thought I would try Then tried building... Since it was based off my main branch it picked up the most recent changes (aagh) and so Then found my test sketches still don't build as to the changes in: #97197 by @ngphibang And figure out what changes I need to make plus see how it will interact with some of the other tests I was playing with earlier. |
It is always easier to make smaller PRs. Currently all reviewers are being quite much solicited as we are approaching a new release and every vendor try to pack their code in place for the new release. Might explain unusual latency in addition to the usual delay. 😬 |
This was extracted from my snapshot PR: zephyrproject-rtos#93797 Been wondering if we should split off from there and at least get the simple recovery mechanism in place? Signed-off-by: Kurt Eckhardt <[email protected]>
I appreciate all that you all do. Question, should I simplify this branch of this PR? Issue a new PR? Or hold off for now and try to push this branch/PR Thanks, Note: @josuah @avolmat-st - I ran a simple sketch (sorry Arduino term - my own sample) that reads image in and displays it With my default of 320x240 on GC2145, I can display a lot of images. It does error, but then recovers, with some
The Dequeue messages are where it has received an image. And you can see the warning about errors. If however I bump this up to 480x320 image then I start getting different DMA errors. At best I might get one image.
Note: I pretty sure I would get these same errors on my other branch associated with this PR. |
In my opinion, because of the number of comments here, open a new PR with the crop support for GC2145 and DCMI with absolute minimal changes to get it working. |
This was extracted from my snapshot PR: zephyrproject-rtos#93797 Been wondering if we should split off from there and at least get the simple recovery mechanism in place? Signed-off-by: Kurt Eckhardt <[email protected]>
Added support to video_stm32_dcmi for the new
set and get selection. These implementations simply forward the message to the underlying
camera object if they support these messages.
Also added support for a snapshot mode instead of
always using continuous capture mode. Tried
to make it semi-transparent when you desire it to be
The stm32_dcmi code now also allows you to work
with only one buffer. This will force it into snap shot mode. There is also new calls added to the api for: video_get_snapshot_mode and video_set_snapshot_mode.
That allows you to set it with more than one buffer and query what mode you are in.
GC2145 was updated first to try out these changes. The camera now allows me to follow the call order
that @josuah mentioned in another pr/issue.
With this driver I also updated it to allow more or less any video resolution:
{
.pixelformat = format, .width_min = width_l, .width_max = width_h,
.height_min = height_l, .height_max = height_h, .width_step = 0, .height_step = 0,
}
When resolution is set, it computes the scale factor. If you then later call set_crop, the same code is
used except it uses the ratios computed from the set_resolution.
With these changes: I was able to setup a test app, for the Arduino Nicla vision and send out a 480x320 image over USB.
More to come
Note: this is a replacement for #91975
My current test sketch/app is up at:
https://github.com/KurtE/zephyr_test_sketches/tree/master/camera_capture_to_usb
built using:
I am using the Arducam viewer with this on my PC. I am using the one at:
https://github.com/mjs513/Teensy_Camera/tree/main/extras/host_app
Picture using GC2145 on Arduino Nicla Vision shown on Arducam mini viewer.

Edit: current summary of changes:
There are several changes some of which will likely change if/when code reviews happen. Things like:
a) The stm_dcmi driver handles the get/set selection APIs and if the camera has also implemented these APIs, it
forwards the messages to them, else return error not implemented.
a1) GC2145 camera implements them.
b) Currently I allow arbitrary size of the frame GC2145, that is I have one fmt (per RGB...) which sets min and max versus the
ones where it current code which has 3 arbitrary sizes (1600x1200 - ratio 1, 640x480 ratio 2, 320x240 ratio 3). I instead
compute ratio and allow you to choose for example 800x600 which is less arbitrary than the 640x480... Note 320x240
computes ratio=5, except I currently limit to max ratio=3 per seeing other implementations that do so... Maybe should
make that max configurable.
c) Setting to allow the code to run in SNAPSHOT mode, which starts the camera, waits for one frame to come back and then deactivates.
This is the way that Arduino library works at least on MBED. Note snapshot mode also has some ability to recover from
failures...
d) Allow you to configure to only have one buffer, before it required at least two, If set to 1, it set forces SNAPSHOT mode.
With this running on Zephyr, I for example was able to program a Nicla Vision, which has no SDRAM and output at
480x320 over USB to an Arducam viewer. My test sketch (not sure what to call them on Zephyr)
is up at https://github.com/KurtE/zephyr_test_sketches/tree/master/camera_capture_to_usb
Also have others that output to Portenta H7 to an ST7796 display...