Charuco seems inaccurate.

[NMO13]

System information (version)

• opencv-python => 4.5.5
• Operating System / Platform => Linux 64 Bit
• Python => 3.9.7

Detailed description

I created a small script here which finds corners of some images containing a chessbaord pattern which I have made, calculates the fundamental matrix and visualizes epipolar lines. I used charuco for finding the corners. The problem: The epipolar lines seem always very off in the visualizations. Therefore I added cv2.findChessboardCorners in order to compare it with the charuco corner detection and found that it seems to be much more accurate when visualizing the epilines.

[classner]

Hi @NMO13 , @alalek ,

A thing I have done in the past is to play around with the corner refinement method. Best results (but most compute) I used to have with the Apriltag refinement. However, that is broken for me in the latest OpenCV release (installed 4.5.5.62 opencv-python and opencv-contrib-python, platform Windows 10, x64). When I use the pattern detector, it hangs indefinitely without returning a result.

[asmorkalov]

Related work: #3174

[asmorkalov]

Related issue with the algorithm hanging: #2816

[AleksandrPanov]

This PR #3174 could fix your issue. Could you attach the reproducer and images?

[asmorkalov]

@AleksandrPanov reproducer and images are located in linked repo. See description.

[AleksandrPanov]

@NMO13 thx for the very detailed description and nice code. There is a problem with object points for ChArUco. You created the chessboard with size (7, 5) and you created the ChArUco board with size = (5, 7) - board = aruco.CharucoBoard_create(5, 7, square_length, marker_length, aruco_dict)
(but in the image, you rotated the ChArUco board by 90 degrees).

// object points order of ChArUco.

// object points order of chessboard.

There are different object points orders for ChArUco and chessboard, in this case. You can use the field chessboardCorners from the ChArUco board to avoid manually creating object points:

charuco_chessboard_corners = board.chessboardCorners
...
for counter in range(int(len(f)/2)):
    ...
    charuco_objpts.append(charuco_chessboard_corners)
    ...
F_charuco = calibrate(charuco_corners_list_left, charuco_corners_list_right, img_left.shape, charuco_objpts)

After the fix, everything works.
With findChessboardCorners() - error 0.686.
With ChArUco - error 0.298.

Also, attached is the working script.
main.py.txt

Do you mind if photos from your repository can be used to improve the ArUco tutorial?

[AleksandrPanov]

Hi @NMO13 , @alalek ,

A thing I have done in the past is to play around with the corner refinement method. Best results (but most compute) I used to have with the Apriltag refinement. However, that is broken for me in the latest OpenCV release (installed 4.5.5.62 opencv-python and opencv-contrib-python, platform Windows 10, x64). When I use the pattern detector, it hangs indefinitely without returning a result.

@classner, could you attach a reproducer? cv::aruco::CORNER_REFINE_APRILTAG is working with these samples.

[NMO13]

@NMO13 thx for the very detailed description and nice code. There is a problem with object points for ChArUco. You created the chessboard with size (7, 5) and you created the ChArUco board with size = (5, 7) - board = aruco.CharucoBoard_create(5, 7, square_length, marker_length, aruco_dict) (but in the image, you rotated the ChArUco board by 90 degrees). // object points order of ChArUco.

// object points order of chessboard.

There are different object points orders for ChArUco and chessboard, in this case. You can use the field chessboardCorners from the ChArUco board to avoid manually creating object points:

charuco_chessboard_corners = board.chessboardCorners
...
for counter in range(int(len(f)/2)):
    ...
    charuco_objpts.append(charuco_chessboard_corners)
    ...
F_charuco = calibrate(charuco_corners_list_left, charuco_corners_list_right, img_left.shape, charuco_objpts)

After the fix, everything works. With findChessboardCorners() - error 0.686. With ChArUco - error 0.298.

Also, attached is the working script. main.py.txt

Do you mind if photos from your repository can be used to improve the ArUco tutorial?

Hi @AleksandrPanov, thanks for your effort and clear explanation!
Of course you can use my pictures.