Depth-color registration parameters are unknown

[floe]

I'm still trying to get the calibration/registration parameters sorted out, but I'm kinda stuck, so I'll try to write down what I think I know and hope that someone can provide some additional pointers...

The intrinsics for the depth camera seem pretty certain - for the two
devices I own, they are:

depth camera intrinsic parameters:
fx 368.096588, fy 368.096588, cx 261.696594, cy 202.522202
depth camera radial distortion coeffs:
k1 0.084061, k2 -0.271582, p1 0.000000, p2 0.000000, k3 0.101907

depth camera intrinsic parameters:
fx 365.402802, fy 365.402802, cx 260.925507, cy 205.594604
depth camera radial distortion coeffs:
k1 0.095575, k2 -0.277055, p1 0.000000, p2 0.000000, k3 0.097539

This is consistent with a camera having square pixels at a resolution of 512*424 (although cy seems a bit off-center to me).

Next are color camera intrinsics - there's a 25-value block which contains the following 5 initial values on both cameras:

1081.37 - might again be focal length in pixels - FOV?
 959.5  - exactly half the x resolution -> optical center x?
 539.5  - exactly half the y resolution -> optical center y?
 863    - not really a clue - perhaps x shift relative to depth cam?
  52    - perhaps y shift relative to depth cam?

About the 20 other values, I have no real clue, although part of them look like they might form a 4x4 transformation matrix...

Any additional comments/ideas?

[floe]

@steppobeck @christiankerl any ideas/comments? :-)

[kohrt]

The data I got from two different devices look similar to yours. So each device has his own ir camera parameters, while the color parameters are the same, at least the first 5. Do the other 20 vales are the same on both of your devices?

I calibrated both sensors to each other on both devices. The rotation between them is nearly the identity matrix and the translation looks like this:
[ -0.05192784012425894, -0.0004530758522097678, 0.0007057198534333861 ]
[ -0.05201715212410899, -0.0002404511811226246, 0.0002945446940294137 ]

The 52 could be the distance on x axis in mm. But I have no clue what the 863 could be. And the last 20 values are pretty small.

[floe]

IIRC the additional 20 values are slightly different on each device, I'll re-check.

Is the exact FOV of the color/depth cams known? FOV can be calculated from intrinsic parameters, so this could be used to double-check the 1081.37 value which I suspect to be focal length.

[kohrt]

These are the values I got from calibration the color sensor:
fx 1060.707250708333, cx 956.354471815484
fy 1058.608326305465, cy 518.9784429882449

They are near to the ones provided by the device. So the first 3 are fx/fy, cx, cy

I also released a toolkit for using the kinect2 with ros. This contains an opencl and a cpu based depth registration and a calibration tool. here. maybe this is of use when implementing the registration directly in libfreenect2.

[floe]

That looks very useful, particularly since you already have an OpenCL-based implementation.

My multiple-view geometry is a bit rusty, what's the full set of parameters needed to translate from one camera space to the other? fx/fy, cx/cy for each camera + a 4x4 transformation matrix, IIRC?

[kohrt]

yes, you need the intrinsic (fx,fy,cx,cy, and the distortion coefficients to rectify the images) and extrinsic (translation and rotation) of the sensors. For each pixel of the depth image you can compute a 3d point and then transform it from depth coordinate system to the color coordinate system. Then you can project it back to the image plane of the color image. this results in a depth image from the perspective of the color camera.

[jeppewalther]

This might be helpful
Mapping depth pixels with color pixels

[floe]

Excellent, very helpful. So to summarize, the steps + required parameters are:

• Undistortion of depth image (depth distortion coefficients k1-3,p1-2, known)
• Backprojection into depth cam 3D space (requires fx,fy,cx,cy for depth camera, known)
• Translation/Rotation into color cam 3D space (requires 3x3 rotation matrix + 3 x translation vector, not yet known)
• Undistortion of color image (color distortion coefficients k1-3,p1-2, unknown)
• Projection onto color image plane (requires fx,fy,cx,cy for color camera, known)

So if we assume a regular 4x4 transformation matrix (16 values) for the 3rd step and 5 distortion coefficients for the 4th step, there's a total of 21 unknown values.

[jeppewalther]

I can confirm that the undistortion and backprojection parameters works very well.

[floe]

@jeppewalther good to know! With which codebase did you try this, and what values did you use for translation/rotation?

[jeppewalther]

As i mentioned i can only confirm Undistortion and Backprojection. So i did not use the translation/rotation parameters. If you find numbers for these parameters i will be happy to try them out.

I am doing a univercity project using PCL which i have hooked up with OpenKinect/libfreenect2. The codebase is not public at the moment.

The undistortion is surprisingly important as seen in the image below. Where undistortion have been applied to the x,y pixel coordinates.

Code for creating undistorted x,y cords using OpenCV

int width = 512;
int height = 424;

cv::Mat cv_img_cords = cv::Mat(1, width*height, CV_32FC2);
for (int r = 0; r < height; ++r) {
    for (int c = 0; c < width; ++c) {
        cv_img_cords.at<cv::Vec2f>(0, r*width + c) = cv::Vec2f((float)r, (float)c);
    }
}

cv::Mat k = cv::Mat::eye(3, 3, CV_32F);
k.at<float>(0,0) = config.camera_params.ir.fx;
k.at<float>(1,1) = config.camera_params.ir.fy;
k.at<float>(0,2) = config.camera_params.ir.cx;
k.at<float>(1,2) = config.camera_params.ir.cy;

cv::Mat dist_coeffs = cv::Mat::zeros(1, 8, CV_32F);
dist_coeffs.at<float>(0,0) = config.camera_params.ir.k1;
dist_coeffs.at<float>(0,1) = config.camera_params.ir.k2;
dist_coeffs.at<float>(0,2) = config.camera_params.ir.p1;
dist_coeffs.at<float>(0,3) = config.camera_params.ir.p2;
dist_coeffs.at<float>(0,4) = config.camera_params.ir.k3;

cv::Mat new_camera_matrix = cv::getOptimalNewCameraMatrix(k, dist_coeffs, cv::Size2i(height,width), 0.0);

cv::undistortPoints(cv_img_cords, cv_img_corrected_cords, k, dist_coeffs, cv::noArray(), new_camera_matrix);

Function for projection into 3D

bool ObjectTracker::depth_image_to_point_cloud(cv::Mat& depth, cv::Mat& xycords, pcl::PointCloud<PointT>::Ptr dst)
{
    //Process Depth To PCL Cloud
    uint pixel_count = depth.rows * depth.cols;
    dst->resize(pixel_count);
    dst->height = depth.rows;
    dst->width = depth.cols;

    float x = 0.0f, y = 0.0f;

    float* ptr = (float*) (depth.data);
    for (uint i = 0; i < pixel_count; ++i)
    {
        cv::Vec2f xy = xycords.at<cv::Vec2f>(0, i);
        x = xy[1]; y = xy[0];

        PointT point;
        point.z = (static_cast<float>(*ptr)) / (1000.0f); // Convert from mm to meters
        point.x = (x - config.camera_params.ir.cx) * point.z / config.camera_params.ir.fx;
        point.y = (y - config.camera_params.ir.cy) * point.z / config.camera_params.ir.fy;
        point.r = 128; point.g = 128; point.b = 128;
        dst->at(i) = point;

        ++ptr;
    }
}

My IR Camera parameters as given by libfreenect2

struct camera_params_t
{
    struct ir_t{
        float cx = 254.878f;
        float cy = 205.395f;
        float fx = 365.456f;
        float fy = 365.456f;
        float k1 = 0.0905474;
        float k2 = -0.26819;
        float k3 = 0.0950862;
        float p1 = 0.0;
        float p2 = 0.0;
    }ir;
}camera_params;

Result

[kohrt]

And from where did you get the translation and rotation? by calibration? And the IR intrinsic are the ones provided by the sensor?
For the rectification you can also use cv::initUndistortRectifyMap and cv::remap. The first one creates a mapping between the undistorted and distorted image, and the second one applies the mapping to an image.

[jeppewalther]

I don´t use translation and rotation for anything. I only need them when working with the RGB camera correct ?. I simply do the coloring based on the IR image.

Ahh now i see the problem .. i am missing a "not" in my previous post.

"So i did use the translation/rotation parameters" should be "So i did not use the translation/rotation
parameters"

Sorry, i will update the previous post.

[jeppewalther]

And yes all the parameters are acquired from the sensor.