Sparse Variable Projection in Robotic Perception: Exploiting Separable Structure for Efficient Nonlinear Optimization

TL;DR

The paper tackles the scalability of large-scale nonlinear least-squares problems in robotic perception by exploiting separable structure and gauge-symmetry considerations through a sparsity-preserving VarPro framework. It introduces a one-time preprocessing step that constructs a matrix-free Schur complement operator, enabling efficient cost, gradient, and Hessian-vector computations within standard iterative solvers. The approach yields substantial runtime reductions (2×–35×) across SLAM, SNL, and SfM benchmarks while preserving accuracy, and includes an open-source C++ implementation and datasets. Together, these contributions offer a practical, scalable path for large perception problems where traditional Schur or direct solvers struggle with memory and conditioning constraints.

Abstract

Robotic perception often requires solving large nonlinear least-squares (NLS) problems. While sparsity has been well-exploited to scale solvers, a complementary and underexploited structure is \emph{separability} -- where some variables (e.g., visual landmarks) appear linearly in the residuals and, for any estimate of the remaining variables (e.g., poses), have a closed-form solution. Variable projection (VarPro) methods are a family of techniques that exploit this structure by analytically eliminating the linear variables and presenting a reduced problem in the remaining variables that has favorable properties. However, VarPro has seen limited use in robotic perception; a major challenge arises from gauge symmetries (e.g., cost invariance to global shifts and rotations), which are common in perception and induce specific computational challenges in standard VarPro approaches. We present a VarPro scheme designed for problems with gauge symmetries that jointly exploits separability and sparsity. Our method can be applied as a one-time preprocessing step to construct a \emph{matrix-free Schur complement operator}. This operator allows efficient evaluation of costs, gradients, and Hessian-vector products of the reduced problem and readily integrates with standard iterative NLS solvers. We provide precise conditions under which our method applies, and describe extensions when these conditions are only partially met. Across synthetic and real benchmarks in SLAM, SNL, and SfM, our approach achieves up to \textbf{2$\times$--35$\times$ faster runtimes} than state-of-the-art methods while maintaining accuracy. We release an open-source C++ implementation and all datasets from our experiments.

Figures (3)

• Figure 1: (Top) method overview: Our approach exploits separability in optimization problem to perform variable projection and analytically eliminate a subset of variables (reducing problem size and improving conditioning), while preserving the efficiency of the original problem's sparsity structure. Our approach can be applied as a one-time preprocessing step before passing the problem to a standard iterative solver. (Bottom) runtime improvement: Comparison of cost vs. time for our reduced problem (green) and the original problem (blue) on a real-world structure from motion dataset (BAL-1934). The variable projection step allows for substantial improvements in solver convergence and overall runtime.
• Figure 2: Matrix-Free Schur Complement Products: Here we demonstrate the difference between explicitly forming the Schur complement $Q_\text{Sc}$ as a dense matrix versus performing a series of sparse operations to implicitly compute the product $Q_\text{Sc} X_{\text{c}}$ without forming the matrix. The top row shows the sparsity pattern of the original matrix $Q$. The bottom two rows show the sparsity patterns of the dense (explicit) and sparse (implicit) Schur complement approaches with the data from the 'Garage' dataset carlone2015Initialization.
• Figure 3: Convergence behavior on select problems. Each panel pair shows (left) cost vs. iterations and (right) cost vs. time for our method and the baseline methodologies on representative datasets chosen from (a) pose-graph optimization (Intel), (b) RA-SLAM (Single Drone), (c) SNL (MIT), and (d) SfM (Mip-NeRF Garden). Notably, the most iteration-efficient method is not always the most time-efficient, as some methods have higher per-iteration costs.