Moving last squares registration in computer vision: new applications and algorithms

Abstract

Registration is a fundamental task in computer vision, and is often used as a preliminary step in diverse applications. In the process of registration, the transformation model needs to be estimated to establish the correspondence relationships between input images. Most transformation models are built upon certain assumptions. However, in practice, when given uncharacteristic data, applying such a model may result in critical deviations/artifacts in the registration output. The research conducted in this thesis focuses on the step of transformation model estimation in registration problems, where the underlying model assumptions do not hold. A central theme of this thesis is the usage of moving least squares (MLS) technique to handle violations to model assumptions. This thesis contributes in three specific applications: radial distortion estimation, image stitching and video stabilization. First, real cameras approximate ideal pinhole cameras using lenses and apertures. This leads to radial distortion effects that are not characterizable by the standard epipolar geometry model and impacts the efficacy of point correspondence validation based on the epipolar constraint. Many previous works deal with radial distortion by augmenting the epipolar geometry model with additional parameters such as distortion coefficients and centre of distortion. In this thesis, radial distortion is treated as a violation to the basic epipolar geometry. To account for the distortion effects, the epipolar geometry is adjusted via the MLS approximation combined with M-estimators to allow robust matching of interest points under severe radial distortion. Compared to previous works, the proposed method is much simpler and exhibits a higher tolerance in cases where the exact model of radial distortion is unknown. Secondly, spatially varying warps are increasingly popular for image alignment as alternatives to homographic warps, since the basic homography model carries the assumptions that images were taken under pure rotational motions, or that the scene is sufficiently far away such that it is effectively planar { conditions unlikely to be satisfied in casual photography. However, estimating spatially varying warps requires a sufficient number of feature matches. In image regions where feature detection or matching fail, the warp loses guidance and is unable to accurately model the true underlying warp, thus resulting in poor registration. This thesis proposes a correspondence insertion method for As-Projective-As-Possible (APAP) warps, which are extensions of MLS to the projective setting. The proposed method automatically identifies misaligned regions, and inserts appropriate point correspondences to increase the flexibility of the warp and improve alignment. Unlike other warp varieties, the underlying projective regularization of APAP warps reduces overfitting and geometric distortion, despite increases to the warp complexity. Lastly, video stabilization is achieved by estimating the camera trajectory throughout the video and then smoothing the trajectory. In practice, most approaches directly model and filter the camera motion using 2D image transforms (e.g., affine or projective). From the smoothed motions, update transforms are obtained to adjust each frame of the video such that the overall sequence appears to be stabilized. However, the update transform is also customarily defined by the basic 2D transforms, which cannot preserve the image contents well. As a result the stabilized videos often appear distorted and “wobbly". Therefore, estimating good update transforms is more critical to success than accurately modeling and characterizing the motion of the camera. Based on this observation, this thesis proposes homography fields for video stabilization. A homography field is a spatially varying warp that is regularized to be as projective as possible, so as to enable accurate warping while adhering closely to the underlying geometric constraints. It has been shown that homography fields are powerful enough to meet the various warping needs of video stabilization, not just in the core step of stabilization, but also in video inpainting. This enables relatively simple algorithms to be used for motion modelling and smoothing. Results on various publicly available testing videos demonstrate the merits of the proposed video stabilization pipeline.