|
| 1 | +# OpenCV pixel-intensity based registration module |
| 2 | + |
| 3 | +Author and maintainer: Alfonso Sanchez-Beato |
| 4 | + alfonsosanchezbeato\_\_\_\_gmail.com |
| 5 | + |
| 6 | +These classes implement a module for OpenCV for parametric image registration. |
| 7 | +The implemented method is direct alignment, that is, it uses directly the pixel |
| 8 | +values for calculating the registration between a pair of images, as opposed to |
| 9 | +feature-based registration. The implementation follows essentially the |
| 10 | +corresponding part of the paper "Image Alignment and Stitching: A Tutorial", |
| 11 | +from Richard Szeliski. |
| 12 | + |
| 13 | +Feature based methods have some advantages over pixel based methods when we are |
| 14 | +trying to register pictures that have been shoot under different lighting |
| 15 | +conditions or exposition times, or when the images overlap only partially. On |
| 16 | +the other hand, the main advantage of pixel-based methods when compared to |
| 17 | +feature based methods is their better precision for some pictures (those shoot |
| 18 | +under similar lighting conditions and that have a significative overlap), due to |
| 19 | +the fact that we are using all the information available in the image, which |
| 20 | +allows us to achieve subpixel accuracy. This is particularly important for |
| 21 | +certain applications like multi-frame denoising or super-resolution. |
| 22 | + |
| 23 | +In fact, pixel and feature registration methods can complement each other: an |
| 24 | +application could first obtain a coarse registration using features and then |
| 25 | +refine the registration using a pixel based method on the overlapping area of |
| 26 | +the images. The code developed allows this use case. |
| 27 | + |
| 28 | +The module implements classes derived from the abstract classes cv::reg::Map or |
| 29 | +cv::reg::Mapper. The former models a coordinate transformation between two |
| 30 | +reference frames, while the later encapsulates a way of invoking a method that |
| 31 | +calculates a Map between two images. Although the objective has been to |
| 32 | +implement pixel based methods, the module could be extended to support other |
| 33 | +methods that can calculate transformations between images (feature methods, |
| 34 | +optical flow, etc.). |
| 35 | + |
| 36 | +Each class derived from Map implements a motion model, as follows: |
| 37 | + |
| 38 | +* MapShift: Models a simple translation |
| 39 | + |
| 40 | +* MapAffine: Models an affine transformation |
| 41 | + |
| 42 | +* MapProject: Models a projective transformation |
| 43 | +MapProject can also be used to model affine motion or translations, but some |
| 44 | +operations on it are more costly, and that is the reason for defining the other |
| 45 | +two classes. |
| 46 | + |
| 47 | +The classes derived from Mapper are |
| 48 | + |
| 49 | +* MapperGradShift: Gradient based alignment for calculating translations. It |
| 50 | +produces a MapShift (two parameters that correspond to the shift vector). |
| 51 | + |
| 52 | +* MapperGradEuclid: Gradient based alignment for euclidean motions, that is, |
| 53 | +rotations and translations. It calculates three parameters (angle and shift |
| 54 | +vector), although the result is stored in a MapAffine object for convenience. |
| 55 | + |
| 56 | +* MapperGradSimilar: Gradient based alignment for calculating similarities, |
| 57 | +which adds scaling to the euclidean motion. It calculates four parameters (two |
| 58 | +for the anti-symmetric matrix and two for the shift vector), although the result |
| 59 | +is stored in a MapAffine object for convenience. |
| 60 | + |
| 61 | +* MapperGradAffine: Gradient based alignment for an affine motion model. The |
| 62 | +number of parameters is six and the result is stored in a MapAffine object. |
| 63 | + |
| 64 | +* MapperGradProj: Gradient based alignment for calculating projective |
| 65 | +transformations. The number of parameters is eight and the result is stored in a |
| 66 | +MapProject object. |
| 67 | + |
| 68 | +* MapperPyramid: It implements hyerarchical motion estimation using a Gaussian |
| 69 | +pyramid. Its constructor accepts as argument any other object that implements |
| 70 | +the Mapper interface, and it is that mapper the one called by MapperPyramid for |
| 71 | +each scale of the pyramid. |
| 72 | + |
| 73 | +If the motion between the images is not very small, the normal way of using |
| 74 | +these classes is to create a MapperGrad\* object and use it as input to create a |
| 75 | +MapperPyramid, which in turn is called to perform the calculation. However, if |
| 76 | +the motion between the images is small enough, we can use directly the |
| 77 | +MapperGrad\* classes. Another possibility is to use first a feature based method |
| 78 | +to perform a coarse registration and then do a refinement through MapperPyramid |
| 79 | +or directly a MapperGrad\* object. The "calculate" method of the mappers accepts |
| 80 | +an initial estimation of the motion as input. |
| 81 | + |
| 82 | +When deciding which MapperGrad to use we must take into account that mappers |
| 83 | +with more parameters can handle more complex motions, but involve more |
| 84 | +calculations and are therefore slower. Also, if we are confident on the motion |
| 85 | +model that is followed by the sequence, increasing the number of parameters |
| 86 | +beyond what we need will decrease the accuracy: it is better to use the least |
| 87 | +number of degrees of freedom that we can. |
| 88 | + |
| 89 | +In the file map_test.cpp some examples on how to use this module can be seen. |
| 90 | +There is a test function for each MapperGrad\*. A motion is simulated on an input |
| 91 | +image and then we register the moved image using a MapperPyramid created with |
| 92 | +the right MapperGrad\*. The difference images of the pictures before and after |
| 93 | +registering are displayed, and the ground truth parameters and the calculated |
| 94 | +ones are printed. Additionally, two images from a real video are registered |
| 95 | +using first SURF features and then MapperGradProj+MapperPyramid. The difference |
| 96 | +between the images and the difference of the registered images using the two |
| 97 | +methods are displayed. It can be seen in the differences shown that using a |
| 98 | +pixel based difference we can achieve more accuracy. |
| 99 | + |
0 commit comments