Feature/aruco speedup #2790

urbste · 2020-12-16T17:16:05Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Hi,

I improved the Aruco module (I will call it Aruco3 as the authors did). It implements the improvements that were proposed in the paper: "Speeded up detection of squared fiducial markers", 2018, FJ Romera-Ramirez et al.

The implementation is not entirely finished, meaning that I would like to know from you guys where I should add tests and so on.
I uploaded simple test videos: https://drive.google.com/drive/folders/153UTdWMOwxIsn2nxdWW0itV8na4AANZH?usp=sharing
The folder contains 3 videos with 4k, 1k and VGA resolution to test the speed improvements and detection rates. However are there any particular tests for the Aruco module to compare against the original implementation?

I added the functionality to the example_aruco_detect_markers.cpp file. In addition, I added new flags to the detector_params.yml. If Aruco3 is turned off, the code should behave as the original Aruco implementation.

# new aruco functionality
useAruco3Detection: 0
minSideLengthCanonicalImg: 32 # 16, 32, 64 --> tau_c from the paper
minMarkerLengthRatioOriginalImg: 0.08 # range [0,0.2] --> tau_i from the paper
cameraMotionSpeed: 0.5 # range [0,1) --> tau_s from the paper
useGlobalThreshold: 0

The parameters are explained in the documentation and reference to equations and sections from the paper.

Some preliminary results for #detected and mean computation per image in [ms]:
Call:

./example_aruco_detect_markers -v=4k_test.mp4 -d=16 -dp=detector_params.yml -refine=1

For all datasets: tau_c = 32pixel

Dataset	Aruco	Aruco3 (tau_i = 0.1)	Aruco3 (tau_i = 0.05)	Aruco3 (tau_i = 0.02)
VGA	921 / 2.91ms	704 / 0.95ms	761 / 1.1ms	909 / 2.1ms
1080p	1186 / 14ms	995 / 2.6ms	1100 / 3.1ms	1180 / 5.3ms
4K	1013 / 63ms	931 / 7.6ms	1011 / 8.1ms	1071 / 13.3ms

For all datasets: tau_c = 16pixel

Dataset	Aruco	Aruco3 (tau_i = 0.1)	Aruco3 (tau_i = 0.05)	Aruco3 (tau_i = 0.02)
VGA	921 / 2.91ms	474 / 0.8ms	796 / 0.9ms	919 / 1.7ms
1080p	1186 / 14ms	132 / 2.9ms	1086 / 2.9 ms	1173 / 4.0 ms
4K	1013 / 63ms	118 / 7.0ms	1013 /7.4ms	1052 / 8.3ms

Now instead of fixing tau_i we can also let the algorithm decide it by using the marker with the smallest perimeter from the last frame. This only works for video streams of course. In addition, we can use global thresholding instead of adaptive thresholding, assuming that the illuminations does not change much.
Call:

./example_aruco_detect_markers -v=4k_test.mp4 -d=16 -dp=detector_params.yml -refine=1 -ar3vid

For all datasets: tau_c = 32pixel, tau_s = 0.1, useGlobalThreshold=1

Dataset	Aruco	Aruco3
VGA	921 / 2.91ms	758 / 1.0ms
1080p	1186 / 14ms	1174 / 2.9ms
4K	1013 / 63ms	1036 / 8.8ms

For all datasets: tau_c = 16pixel, tau_s = 0.1, useGlobalThreshold=1

Dataset	Aruco	Aruco3
VGA	921 / 2.91ms	905 / 0.7ms
1080p	1186 / 14ms	1156 / 2.8ms
4K	1013 / 63ms	1022 / 7.7ms

Let me know what you think.
Cheers
Steffen

urbste · 2020-12-17T06:36:52Z

Ok I can see that already some Aruco tests are failing. I will have a look into it.

…edup

urbste · 2021-02-10T15:19:48Z

I added tests and fixed some small bugs. Can someone check why the build checks are failing? When I click on details it looks like all tests run successfully.

alalek · 2021-02-10T18:15:56Z

You can see compilation warnings here (should be resolved before merge).

alalek

Thank you for the contribution!

Please take a look on comments below.

alalek · 2021-02-11T20:51:59Z

modules/aruco/test/test_charucodetection.cpp

    cameraMatrix.at< double >(1, 2) = imgSize.height / 2;

    Mat distCoeffs(5, 1, CV_64FC1, Scalar::all(0));
+


Please remove unnecessary changes from the patch (changes in this file are not needed)

alalek · 2021-02-11T21:12:28Z

modules/aruco/include/opencv2/aruco.hpp

+ *   constant and global instead of adaptive thresholding can be applied, speeding up the binarization step.
+ * - foundGlobalThreshold: internal variable. It is used to cache the variable to the next detector call.
+ * - otsuGlobalThreshold: internal variable. It is used to cache the global otsu threshold to the next detector call.
+ * - foundMarkerInLastFrames: internal variable. It is used to cache if markers were found in the last frame.


internal variable

Parameters structs are used in OpenCV as non-modified inputs only.
This requirement is necessary for Python and other bindings.

For such video processing case with internal state it make sense to create dedicated API with detector class which store this internal state (use PImpl pattern, see QRCodeDetector class from objdetect module)

alalek · 2021-02-11T21:18:09Z

modules/aruco/src/aruco.cpp

-	return Vec2f(A.solve(B).val);
+    Matx22f A(nLine1.x, nLine1.y, nLine2.x, nLine2.y);
+    Vec2f B(-nLine1.z, -nLine2.z);
+    return Vec2f(A.solve(B).val);


Please avoid such massive space-only changes.
It is really hard to see/track the significant changes of PRs.

Consider moving these "whitespace" changes to the separate first commit or separate PR.

asenyaev · 2021-04-09T16:27:03Z

jenkins cn please retry a build

joellyons · 2021-08-02T21:28:22Z

What's the latest on this PR? Has it been abandoned?

urbste · 2021-08-03T06:09:45Z

I am still having it in the back of my head. However, the requested changes seem to be quite significant and I did not find the time to look at it. We were using this branch in a prototype, however, we moved away from it. So if someone likes to take over please go ahead!

joellyons · 2021-08-03T15:50:24Z

Thank you for the update @urbste . If you don't mind a bit more info... why did you move away from it? Were you having problems with it?

urbste · 2021-08-04T06:41:39Z

@joellyons no we did not have problems with it. We wanted to use it in WebAssembly running on a smartphone to track the pose in realtime. However our board had a lot of markers and we never reached >30fps on Android devices so the AR animations were pretty bumpy. So we kind of changed our algorithm and now only occasionally update the pose using the ARuco board. The Aruco3 algorithm however really shines if you want to track the board over time, so you can update the tau_i parameter and use the global otsu thesholding. So we decided to stick with the classic Aruco implementation.

Still with my PR it is possible to use the classic impementation. I will see if I find the time to finish this PR :/

joellyons · 2021-08-04T13:52:30Z

@urbste thank you so much for taking the time to explain. We are wanting to track a small board over time, so this sounds perfect. But, I agree it's hard to find extra time to work on open source projects like this.

AleksandrPanov · 2021-11-19T08:38:40Z

Hello @urbste, thx for the contribution, your results look great. Don't you have time to finish your PR yet?

urbste added 4 commits December 16, 2020 13:44

initial commit of new aruco functionality

6824448

added example and changed detector params

b6fbfba

fixed a bug

c13af3c

uncomment imshow in example

1820823

urbste added 7 commits December 29, 2020 07:33

merge upstream

1a9d859

fixed bug after merge

5153e76

fixed candidate identification if aruco3=false

bdf139a

Merge remote-tracking branch 'upstream/master' into feature/aruco_spe…

d229ad6

…edup

added test and fixed a bug

c7fdd76

bugfix

927653e

unused parm

ede9777

urbste added 3 commits February 11, 2021 07:27

first fix win compiler warnings

8e91d26

second try eliminate warnings

234b223

one last warning

8a9e7c5

alalek reviewed Feb 11, 2021

View reviewed changes

asmorkalov requested a review from AleksandrPanov December 7, 2021 08:33

This was referenced Jan 19, 2022

Feature/aruco speedup AleksandrPanov/opencv_contrib#1

Closed

Feature/aruco speedup (rebased with 4.x) #3151

Merged

		cameraMatrix.at< double >(1, 2) = imgSize.height / 2;

		Mat distCoeffs(5, 1, CV_64FC1, Scalar::all(0));

Feature/aruco speedup #2790

Feature/aruco speedup #2790

Uh oh!

Conversation

urbste commented Dec 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

urbste commented Dec 17, 2020

Uh oh!

urbste commented Feb 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalek commented Feb 10, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

alalek Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Feb 11, 2021

Choose a reason for hiding this comment

Uh oh!

asenyaev commented Apr 9, 2021

Uh oh!

joellyons commented Aug 2, 2021

Uh oh!

urbste commented Aug 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joellyons commented Aug 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

urbste commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joellyons commented Aug 4, 2021

Uh oh!

AleksandrPanov commented Nov 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

urbste commented Dec 16, 2020 •

edited

Loading

urbste commented Feb 10, 2021 •

edited

Loading

urbste commented Aug 3, 2021 •

edited

Loading

joellyons commented Aug 3, 2021 •

edited

Loading

urbste commented Aug 4, 2021 •

edited

Loading