ccalib/omnidir undistortPoints result differs from undistortImage.

[ftaralle]

System information (version)

• OpenCV => 3.3.0-dev
• both in C++ and Python3

Detailed description

We are using omnidir module to correct wide angle fisheye effect on image data.
We first used cv::omnidir::calibrate to get a valid camera model.
We then used this model to undistort both images and points.

We found that applying undistortion methods with exact same parameters on images and points gives different results.
After investigation we think a bug exists in the undistortPoints method.

Steps to reproduce

Code to reproduce (fixed to be working, thanks to @whplh).

# import of modules
import numpy as np
import cv2
import matplotlib.pyplot as plt
#%matplotlib inline

# in/out image size
height, width = (1520, 2048)

# definition of camera model's parameters
K = np.array([[922.676, -6.87115, 1028.38], [0, 921.053, 718.469], [0, 0, 1]], np.float)
D = np.array([[-0.32628, 0.117317, 0.00124854, 0.000268858]], np.float)
X = np.array([[0.772963]], np.float)

# exposes K parameters
(fx, s, cx), (_, fy, cy), _ = K

# Build a list of 'known' distorted points
nPointsPerCirle = 20
fisheyeRadius = 700
circleRadiuses  = range(0, fisheyeRadius, 100)
angle = 2*np.pi * np.array([*range(nPointsPerCirle)], np.float) / nPointsPerCirle
distorted_points = np.vstack((
    np.vstack((cx + r * np.cos(angle), cy + r * np.sin(angle))).T
    for r in circleRadiuses    
))

# Build a distorted image with that known points on it
distorted_frame = np.zeros((height, width, 3), np.uint8)
cv2.circle(distorted_frame, (int(cx), int(cy)), fisheyeRadius, (255,255,255), -1)
# place points
for distorted_point in distorted_points.astype(int):
    cv2.circle(distorted_frame, tuple(distorted_point), 5, (0,0,255), -1)

# undistort image using omnidir
undistorted_frame = cv2.omnidir.undistortImage(distorted_frame, K, D, X, 1, np.eye(3))

# undistort points 
undistorted_points = cv2.omnidir.undistortPoints(
    np.array([distorted_points.tolist()], np.float), 
    K, D, X, np.eye(3))
x,y = undistorted_points.reshape(-1,2).T
undistorted_points = K.dot(np.vstack((x,y,np.ones_like(x)))).T[:,:2]

# print undistorted points ontop of undistorted image
for undistorted_point in undistorted_points.astype(int):
    cv2.circle(undistorted_frame, tuple(undistorted_point), 5, (255,0,0), 3)

# show result
out_frame = np.hstack((distorted_frame, undistorted_frame))
plt.figure(figsize=(np.array(out_frame.shape[:2]) / 100).tolist())
plt.imshow(cv2.cvtColor(out_frame, cv2.COLOR_BGR2RGB))
plt.axis('off')
plt.show()

[whplh]

Hey there, I've also been working partly with the omnidir module during the last months.
(@ftaralle your code doesn't work properly btw.)
Anyway lets discuss the model implemented in the opencv omnidir module and lets help each other understanding it. The documentation is not too clear about all the details.

1. To my understanding the unified projection model in the version suggested by Mei2007 (Single View Point Omnidirectional Camera Calibration from Planar Grids) is generally usable for catadioptric systems.
   In the special case for xi=1 the model relates spherical coordinates with a planar projection (see Mei 2007 for details). This relation "can cover some existing models for fisheye cameras and fit well for many actual fisheye cameras" ( see https://doi.org/10.1007/978-3-540-24670-1_34). For xi=1 the model is equivalent to the previously found "division model" (https://ieeexplore.ieee.org/document/990465/).

2. So if you fix xi to 1 in your calibration, you will get parameters which correspond to a physical model behind it, meaning you will get the parameters for the generalized camera matrix (Mei 2007), the radial and tangential distortion coefficients, and your model will correspond somewhat to the division model.

3. So I'm not sure if this is a bug here, because if you do the calibration with fixed xi=1.0 you will see that the undistorted points match the undistorted image much better. If you don't fix xi you will just get some nonlinearly optimized coefficients which fit your dataset - this is how I understand it - and I'm not sure if that has some relevance for any model behind it.

4. FYI - the unified projection model in the version proposed by Scaramuzza ( "A Toolbox for Easily Calibrating Omnidirectional Cameras") is apparently much better suited for fisheye cameras (see Matlab implementation/doc). It would be great to find it in OpenCV ;)

To conclude, if anyone with better understanding of the model can clarify the note I made in 3) or any other remark concerning fisheye calibration with the omnidir module, it would be great!
Maybe @jiuerbujie or @prclibo can help understanding the module in more detail here.
Cheers

@ftaralle here is the working version of your code with xi=1 . Maybe you can update K,D based on the calibration data

import cv2
# import of modules
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# in/out image size
height, width = (1520, 2048)

# definition of camera model's parameters
K = np.array([[922.676, -6.87115, 1028.38], [0, 921.053, 718.469], [0, 0, 1]], np.float)
D = np.array([[-0.32628, 0.117317, 0.00124854, 0.000268858]], np.float)
X = np.array([[1.0]], np.float)

# exposes K parameters
(fx, s, cx), (_, fy, cy), _ = K

# Build a list of 'known' distorted points
nPointsPerCirle = 20
fisheyeRadius = 700
circleRadiuses  = range(0, fisheyeRadius, 100)
angle = 2*np.pi * np.array([range(nPointsPerCirle)], np.float) / nPointsPerCirle
distorted_points = np.vstack((
    np.vstack((cx + r * np.cos(angle), cy + r * np.sin(angle))).T
    for r in circleRadiuses    
))

# Build a distorted image with that known points on it
distorted_frame = np.zeros((height, width, 3), np.uint8)
cv2.circle(distorted_frame, (int(cx), int(cy)), fisheyeRadius, (255,255,255), -1)
# place points
for distorted_point in distorted_points.astype(int):
    cv2.circle(distorted_frame, tuple(distorted_point), 5, (0,0,255), -1)

# undistort image using omnidir
undistorted_frame = cv2.omnidir.undistortImage(distorted_frame, K, D, X, 1, np.eye(3))

# undistort points 
undistorted_points = cv2.omnidir.undistortPoints(
    np.array([distorted_points.tolist()], np.float), 
    K, D, X, np.eye(3))
x,y = undistorted_points.reshape(-1,2).T
undistorted_points = K.dot(np.vstack((x,y,np.ones_like(x)))).T[:,:2]

# print undistorted points ontop of undistorted image
for undistorted_point in undistorted_points.astype(int):
    cv2.circle(undistorted_frame, tuple(undistorted_point), 5, (255,0,0), 3)

# show result
out_frame = np.hstack((distorted_frame, undistorted_frame))
plt.figure(figsize=(np.array(out_frame.shape[:2]) / 100).tolist())
plt.imshow(cv2.cvtColor(out_frame, cv2.COLOR_BGR2RGB))
plt.axis('off');  

[ftaralle]

Hi @whplh. Thanks for your helping comment.
I first fixed the code example, mistakes where made importing from a jupyter notebook. -_-

Using X=1 seems strange to me. Indeed what you propose works fine in foreward / backward projection. But it looks like a workaround more than a fix. 🤔

Thus, I investigated more closely X's usage, in omnidir's source code; more precisely the cv::omnidir::undistortPoints

Line 327, undistorted projection point is projected from Unit-Sphere to Unit-Plan (Z=1):

Vec3d ppu = Vec3d(
    Xs[0]/(Xs[2]+_xi), 
    Xs[1]/(Xs[2]+_xi), 
    1.0
);

I wonder why _xi is used here. To me, it should solely be:

Vec3d ppu = Vec3d(
    Xs[0]/Xs[2], 
    Xs[1]/Xs[2], 
    1.0 // <= Xs[2]/Xs[2]
);

I tested with this modified omnidir code. It seems to work fine now: no more discrepency between image back-projection and points back-projection.

So, is this a bug in omnidir or a misunderstanding of me ?

[whplh]

Well it would be helpful to understand why your version works. It would be the same as xi=0. Bear in mind that according to the unified projection model you are not exactly just projecting from a sphere to a plane.

During the projection from the sphere to the plane there is a shift of the reference frame (See Mei's paper). Your suggestion would omit this shift, which doesn't quite fit to the given model.

[ftaralle]

Hi,
To me it's not using xi=0, since it is used in projecting Pi (undistorted point on image plan) to Ps (point on the unit sphere). But i'm wondering why using xi to convert Ps into Pu (point on the unit plan) since by definition unit plan is tangential to unit sphere.

Here is how I understand the model:

[hushunda]

I had a problem.

System information (version)
python 3.6
opencv-contrib-python (4.2.0.32)
opencv-python (4.2.0.32)

Detailed description
We are using omnidir module to correct wide angle fisheye and get Internal parameters (K,D,xi) and external parameters(rvec,tvec) .
But when using the the same images and the Internal reference (K,D,xi) to get undistort points, and using cv2.solvePnP(objPoints, undistorted, np.eye(3), np.zeros([4,1])) to calculate external parameters ,the result are difference from before

[rgolovanov]

@ftaralle the fix now in master. Do you think you need to try it before closing PR?

[mrtranducdung]

Hi, i am looking for a function to convert back a point in undistorted image back to the original distorted points. I know a function for fisheye camera like cv.fisheye.distortPoints, But i cannot find one for omnicamera. Please anyone here know the function?

[wtyuan96]

@ftaralle the fix now in master. Do you think you need to try it before closing PR?

@ftaralle @rgolovanov

I have thoroughly examined the omnidir source code and the documentation corresponding to the undistortPoints function (pay attention to the exact wording "using CMei's model" in the documentation.). I offer an alternative perspective, suggesting that the original author may have intentionally written it this way (with xi), because the undistortPoints function is designed for undistortion using Mei model, instead of general projection model.