Bundle adjustment

A sparse matrix obtained when solving a modestly sized bundle adjustment problem. This is the arrowhead sparsity pattern of a 992×992 normal-equation (i.e. approximate Hessian) matrix. Black regions correspond to nonzero blocks.

In photogrammetry and computer stereo vision, bundle adjustment is simultaneous refining of the 3D coordinates describing the scene geometry, the parameters of the relative motion, and the optical characteristics of the camera(s) employed to acquire the images, given a set of images depicting a number of 3D points from different viewpoints. Its name refers to the geometrical bundles of light rays originating from each 3D feature and converging on each camera's optical center, which are adjusted optimally according to an optimality criterion involving the corresponding image projections of all points.

Uses

Bundle adjustment is almost always used as the last step of feature-based 3D reconstruction algorithms. It amounts to an optimization problem on the 3D structure and viewing parameters (i.e., camera pose and possibly intrinsic calibration and radial distortion), to obtain a reconstruction which is optimal under certain assumptions regarding the noise pertaining to the observed image features: If the image error is zero-mean Gaussian, then bundle adjustment is the Maximum Likelihood Estimator. Bundle adjustment was originally conceived in the field of photogrammetry during the 1950s and has increasingly been used by computer vision researchers during recent years.

General approach

Bundle adjustment boils down to minimizing the reprojection error between the image locations of observed and predicted image points, which is expressed as the sum of squares of a large number of nonlinear, real-valued functions. Thus, the minimization is achieved using nonlinear least-squares algorithms. Of these, Levenberg–Marquardt has proven to be one of the most successful due to its ease of implementation and its use of an effective damping strategy that lends it the ability to converge quickly from a wide range of initial guesses. By iteratively linearizing the function to be minimized in the neighborhood of the current estimate, the Levenberg–Marquardt algorithm involves the solution of linear systems termed the normal equations. When solving the minimization problems arising in the framework of bundle adjustment, the normal equations have a sparse block structure owing to the lack of interaction among parameters for different 3D points and cameras. This can be exploited to gain tremendous computational benefits by employing a sparse variant of the Levenberg–Marquardt algorithm which explicitly takes advantage of the normal equations zeros pattern, avoiding storing and operating on zero-elements.

Mathematical definition

Bundle adjustment amounts to jointly refining a set of initial camera and structure parameter estimates for finding the set of parameters that most accurately predict the locations of the observed points in the set of available images. More formally, assume that [math]\displaystyle{ n }[/math] 3D points are seen in [math]\displaystyle{ m }[/math] views and let [math]\displaystyle{ \mathbf{x}_{ij} }[/math] be the projection of the [math]\displaystyle{ i }[/math]th point on image [math]\displaystyle{ j }[/math]. Let [math]\displaystyle{ \displaystyle v_{ij} }[/math] denote the binary variables that equal 1 if point [math]\displaystyle{ i }[/math] is visible in image [math]\displaystyle{ j }[/math] and 0 otherwise. Assume also that each camera [math]\displaystyle{ j }[/math] is parameterized by a vector [math]\displaystyle{ \mathbf{a}_j }[/math] and each 3D point [math]\displaystyle{ i }[/math] by a vector [math]\displaystyle{ \mathbf{b}_i }[/math]. Bundle adjustment minimizes the total reprojection error with respect to all 3D point and camera parameters, specifically

[math]\displaystyle{ \min_{\mathbf{a}_j, \, \mathbf{b}_i} \displaystyle\sum_{i=1}^{n} \; \displaystyle\sum_{j=1}^{m} \; v_{ij} \, d(\mathbf{Q}(\mathbf{a}_j, \, \mathbf{b}_i), \; \mathbf{x}_{ij})^2, }[/math]

where [math]\displaystyle{ \mathbf{Q}(\mathbf{a}_j, \, \mathbf{b}_i) }[/math] is the predicted projection of point [math]\displaystyle{ i }[/math] on image [math]\displaystyle{ j }[/math] and [math]\displaystyle{ d(\mathbf{x}, \, \mathbf{y}) }[/math] denotes the Euclidean distance between the image points represented by vectors [math]\displaystyle{ \mathbf{x} }[/math] and [math]\displaystyle{ \mathbf{y} }[/math]. Because the minimum is computed over many points and many images, bundle adjustment is by definition tolerant to missing image projections, and if the distance metric is chosen reasonably (e.g., Euclidean distance), bundle adjustment will also minimize a physically meaningful criterion.

See also

• Adjustment of observations
• Stereoscopy
• Levenberg–Marquardt algorithm
• Sparse matrix
• Collinearity equation
• Structure from motion
• Simultaneous localization and mapping

Further reading

• A. Zisserman. Bundle adjustment. CV Online.

External links

Software

• [1]: Apero/MicMac, a free open source photogrammetric software. Cecill-B licence.
• sba: A Generic Sparse Bundle Adjustment C/C++ Package Based on the Levenberg–Marquardt Algorithm (C, MATLAB). GPL.
• cvsba : An OpenCV wrapper for sba library (C++). GPL.
• ssba: Simple Sparse Bundle Adjustment package based on the Levenberg–Marquardt Algorithm (C++). LGPL.
• OpenCV: Computer Vision library in the Images stitching module. BSD license.
• mcba: Multi-Core Bundle Adjustment (CPU/GPU). GPL3.
• libdogleg: General-purpose sparse non-linear least squares solver, based on Powell's dogleg method. LGPL.
• ceres-solver: A Nonlinear Least Squares Minimizer. BSD license.
• g2o: General Graph Optimization (C++) - framework with solvers for sparse graph-based non-linear error functions. LGPL.
• DGAP: The program DGAP implement the photogrammetric method of bundle adjustment invented by Helmut Schmid and Duane Brown. GPL.
• Bundler: A structure-from-motion (SfM) system for unordered image collections (for instance, images from the Internet) by Noah Snavely. GPL.
• COLMAP: A general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline with a graphical and command-line interface. BSD license.
• Theia: A computer vision library aimed at providing efficient and reliable algorithms for Structure from Motion (SfM). New BSD license.
• Ames Stereo Pipeline has a tool for bundle adjustment (Apache II licence).