MinBackProp -- Backpropagating through Minimal Solvers

TL;DR

This work tackles the challenge of backpropagating through minimal solvers within end-to-end learning by leveraging the Implicit Function Theorem (IFT) to compute derivatives of solver outputs with respect to inputs. The authors provide a direct IFT-based backpropagation method and contrast it with a Deep Declarative Networks (DDN) approach, showing that IFT is simple, fast, and numerically stable, while DDN offers an easier but slower alternative. They validate the approach on a toy 3D point-registration task with outliers and a real epipolar geometry estimation pipeline, reporting 100% stability and about 10x speed improvements over autograd baselines. The work also details the theoretical underpinnings, including differentiation of polynomial systems and the use of DDN for backprop through algebraic solvers, highlighting its practical impact for robust, differentiable minimal solvers in computer vision tasks such as essential matrix estimation and RANSAC-based sampling.

Abstract

We present an approach to backpropagating through minimal problem solvers in end-to-end neural network training. Traditional methods relying on manually constructed formulas, finite differences, and autograd are laborious, approximate, and unstable for complex minimal problem solvers. We show that using the Implicit function theorem (IFT) to calculate derivatives to backpropagate through the solution of a minimal problem solver is simple, fast, and stable. We compare our approach to (i) using the standard autograd on minimal problem solvers and relate it to existing backpropagation formulas through SVD-based and Eig-based solvers and (ii) implementing the backprop with an existing PyTorch Deep Declarative Networks (DDN) framework. We demonstrate our technique on a toy example of training outlier-rejection weights for 3D point registration and on a real application of training an outlier-rejection and RANSAC sampling network in image matching. Our method provides $100\%$ stability and is 10 times faster compared to autograd, which is unstable and slow, and compared to DDN, which is stable but also slow.

Figures (4)

• Figure 1: A typical end-to-end training pipeline with a minimal problem Solver Wei_2023_ICCV that trains a Neural Network (NN) to predict correct matches. Forward pass: Tentative handcrafted matches $m$ between images $I_1, I_2$ are clarified using NN zhao2021progressive parametrized by trained weights $\theta$, and scores $s$ for these matches are computed. Differentiable D-RANSAC selects a minimal data sample $w$ using the scores $s$; the Solver computes a model $\hat{E}$, which is scored by the loss $J$ using the correct matches $m(i_{gt})$ with the ground truth inlier indicator $i_{gt}$. The groundtruth $E_{gt}$ is passed to the Solver to choose the closest model. Backward pass: Gradient $\frac{\partial J}{\partial \theta}$ for training weights $\theta$ is computed by the chain rule. The key issue is robustly and efficiently backpropagating through the Solver. We propose to use the Implicit function theorem (IFT) directly or implement the backpropagation via the PyTorch Deep Declarative Network (DDN) machinery Gould2022.
• Figure 2: The figure demonstrates the backpropagation for the toy example for the 3D point registration with an outlier. The forward pass is the Kabsch algorithm Kabsch:a12999 (SVD) and remains the same for both explicit and implicit methods. The backward pass is performed explicitly via the closed-form gradients of SVD Ionescu2015 (a) and implicitly, using the Deep Declarative Networks (DDN) Gould2022 and the Implicit function theorem (IFT) (b).
• Figure 3: A toy example for the 3D point registration problem with an outlier: (a) visualization of four points before and after rotation $\mathop{\mathrm{\mathrm{R}}}\nolimits$ with inlier/outlier weights $\mathrm{w}$, blue points denote inliers, and the magenta point is the outlier; (b) the weight for the outlier (left) and the weight for an inlier (right) during the optimization. The red color indicates explicit computation of the gradients, the blue color denotes backpropagation with the DDN, and the green color is backpropagation with the IFT. One can see that the gradients computed with the DDN and the IFT approximate the gradients of SVD computed in closed form.
• Figure 4: A toy example for the fundamental matrix estimation with an outlier: (a) the weight for the outlier and (b) the weight for an inlier during the optimization. The red indicates explicit computation of the gradients, and the green/blue denotes backpropagation with our MinBackProp. One can see that the gradients computed with the Implicit Function theorem and the Deep Declarative Networks within our least squares formulation with constraints \ref{['eq:supp:fundamental']} approximate the gradients of SVD computed in closed form.

Theorems & Definitions (7)

• Theorem 1: Differentiating roots of a polynomial system
• proof
• Example 1
• Theorem 1: Differentiating roots of a polynomial system
• Theorem 2: Implicit function theorem
• proof
• proof