Charuco board detection is not suitable for calibration?

[victor1234]

System Information

OpenCV python version: 4.7.0
Operating System / Platform: Debian GNU/Linux 11 (bullseye)
Python version: 3.9.2

Detailed description

It's my first time using charuco and I did not expect to encounter so many problems with this method.

According to the OpenCV docs, the idea of introducing Charuco was to make the classic chessboard pattern robust to partial view.

But when I ran the charuco corners detection, I realized that the detected chessboard corners did not match the real ones

According to the documentation and source code of cv::CharucoDetector::detectBoard(), this function doesn't detect chessboard corners. Instead, it detects inner ArUco markers and uses them to estimate the camera pose and project back the chessboard corners. It has two options: when you have camera intrinsic parameters and when you don't (it's our option). For the second option, it uses a homography matrix to estimate the chessboard corners, which means that this method does not account for distortion. Therefore, the reprojected positions of corners may be incorrect. After that, the method calls the cv::cornerSubPix() method to refine the corners' positions, and if the initial reprojection position is not far away from the real one, this call could produce accurate detection. That's why not all corners are found incorrectly.

Although this method is mentioned in many tutorials, it is not suitable for calibration.

Steps to reproduce

I got this problem running my own code, but for baseline I ran the official example

./example_aruco_calibrate_camera_charuco -v=<...>/frame-%03d.png -d 0 -h=6 -w=4 --sl=0.3175 --ml=0.2540 --sc --rs

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[qq978358810]

66666

[AleksandrPanov]

@viktor1234 could you attach the original sample image?

[victor1234]

[asmorkalov]

@victor1234 Thanks a lot for the issue report! It's definitely critical issue for ChAruco detector and calibration. Will try to solve it ASAP. Also big thanks for the original image! Could you provide ChaRuco board parameters: dictionary, geometry parameters, etc?

[victor1234]

"markerLength": 0.2540,
"squareLength": 0.3175,
"numberOfMarkersX": 4,
"numberOfMarkersY": 6,
"dictionary": "DICT_4X4_50",

You are welcome.
But Charuco calibration has been in many tutorials and examples for years. I could not understand how it was possible to miss this error. What was the original idea of using reprojected(!) corner positions for calibration?
Low unit test coverage makes OpenCV too risky to use even for prototyping

[asmorkalov]

The code was migrated from opencv_contrib and we are actively working on its stabilization.

[AleksandrPanov]

@victor1234, I found the problem. You are using the deprecated ChArUco template. Just add this code to fix your problem:

board.setLegacyPattern(True)

Charuco templates was updated in opencv/opencv_contrib#3174. The behavior has changed only for templates with an even number of rows. This was done to make the ChArUco patterns compatible with the chessboard.
Use gen_pattern.py to create new pattern or use setLegacyPattern.

Also OpenCV team tested CharucoDetector with another camera:

The results were also quite good. Nevertheless, we need to make a big comparison of findChessboardCorners vs CharucoDetector.
If the CharucoDetector does not work well enough, the OpenCV team can try to improve the algorithm using cornerHarris or other methods. But we need a large dataset and a community request.

[AleksandrPanov]

The script to generate the ChArUco board:

cols = 4
rows = 6
square_size = 100
marker_size = 0.8*square_size
aruco_dict = cv.aruco.getPredefinedDictionary(0) # cv.aruco.DICT_4X4_250
board = cv.aruco.CharucoBoard((cols, rows), square_size, marker_size, aruco_dict)
board.setLegacyPattern(True) # comment this line to create the new template
from_cv_img = board.generateImage((cols*square_size, rows*square_size))
cv.imshow("board_6_4_dict_0", from_cv_img)
cv.imwrite("board_6_4_dict_0.png", from_cv_img)

The deprecated ChArUco template:

The new ChArUco template (this template is compatible with the chessboard):