SDPRLayers: Certifiable Backpropagation Through Polynomial Optimization Problems in Robotics

TL;DR

This work tackles the vulnerability of differentiable optimization in robotics to convergence on spurious local minima, which can distort backpropagated gradients. It introduces SDPRLayers, a differentiable optimization layer that solves semidefinite relaxations of polynomial optimization problems (POPs) and, when the relaxation is tight, yields gradients corresponding to the globally optimal solution through implicit differentiation and certificate matrices. The authors provide theoretical conditions for when gradients are certifiably correct, describe multiple backpropagation schemes (implicit selections, classic IFT, and SDP-based differentiation), and demonstrate strong performance on synthetic polynomial problems, stereo localization, and a deep-learning–aided localization pipeline. The results show that relying on SDP-based gradients avoids the pitfalls of local minima, improves convergence reliability, and enables robust end-to-end learning in robotics, with open-source PyTorch tooling for practical adoption.

Abstract

A recent set of techniques in the robotics community, known as certifiably correct methods, frames robotics problems as polynomial optimization problems (POPs) and applies convex, semidefinite programming (SDP) relaxations to either find or certify their global optima. In parallel, differentiable optimization allows optimization problems to be embedded into end-to-end learning frameworks and has received considerable attention in the robotics community. In this paper, we consider the ill effect of convergence to spurious local minima in the context of learning frameworks that use differentiable optimization. We present SDPRLayers, an approach that seeks to address this issue by combining convex relaxations with implicit differentiation techniques to provide certifiably correct solutions and gradients throughout the training process. We provide theoretical results that outline conditions for the correctness of these gradients and provide efficient means for their computation. Our approach is first applied to two simple-but-demonstrative simulated examples, which expose the potential pitfalls of reliance on local optimization in existing, state-of-the-art, differentiable optimization methods. We then apply our method in a real-world application: we train a deep neural network to detect image keypoints for robot localization in challenging lighting conditions. We provide our open-source, PyTorch implementation of SDPRLayers.

Figures (7)

• Figure 1: Our SDPRLayer embedded in a PyTorch autodifferentiation graph. For non-convex problems with tight semidefinite relaxations, the SDPRLayer finds the certified, globally optimal solution and makes it differentiable via implicit differentiation of the non-convex QCQP. Current differentiable solvers for non-convex problems can return gradients of spurious local minima rather than the gradients of the global solution, corrupting the learning process. In contrast, the gradients produced by the SDPRLayer correspond to the global solution as long as the relaxation is tight, leading to better training. Even when the relaxation is not tight, the relaxed, SDP solution is still provided, and can be used to find good initializations for local methods. The relaxed solution is also made differentiable by leveraging existing differentiable convex optimization layers.
• Figure 2: Evolution of the polynomial function throughout the bilevel optimization. Columns present different methods for solving the inner optimization problem: (left) gradient descent initialized at $x=2$, (center) gradient descent initialized at $x=-2$, and (right) our approach. In all three cases, the colour of the function indicates the progress of outer loop iterations (blue at the beginning, red at the end), the black dots indicate the minima found by the inner optimization, and the plus sign indicates the target. All three cases converge, but gradient descent only converges to a valid solution when it is initialized well ($x_0=-2$) and, even then, is temporarily trapped in a local minimum. SPDRLayer converges to a valid solution, does not require initialization, and is even able to discontinuously 'switch' the minimum to a new region.
• Figure 3: a. Setup for one of the ground-truth poses used in the stereo tuning example. The position of the ground-truth poses are placed 3 m from the center of the grid in a cone of angle $\beta = 90~\hbox{deg}$ (red cone). The poses are oriented such that the center of the grid is within the field of view (FOV) of the camera ($z$) axis, $\alpha=90~\hbox{deg}$. b. 100 random initialization samples for Theseus' Gauss-Newton solver. The initializations are colour coded based on whether they converged to the local minimum (red) or global minimum (green). The local and global solutions are orange and magenta frames, respectively.
• Figure 4: Baseline parameter error trajectories across outer optimization iterations for 50 trials with different inner loop optimization approaches. Theseus with ground-truth initialization and SDPRLayer always converge to global minima, hence provide correct gradients to the outer opt. Theseus with random initialization sometimes converges to local minima and therefore provides incorrect gradients to the outer optimization, resulting in large parameter error.
• Figure 5: Diagram of our Pytorch pipeline (based on the pipeline in gridsethKeepingEyeThings2022) that we use to estimate the relative pose between two images (i.e., localize a target image to a (stored) source image). Orange blocks denote the deep neural networks whose weights are tuned during training (VGG16 network and UNet Decoder network). Blue blocks denote blocks that are part of the pipeline, but do not have trainable parameters. Red blocks indicate key PyTorch tensor variables. Green blocks and arrows indicate training loss function computation.

Theorems & Definitions (8)

• Definition 1: Local Selection
• Theorem 2: Implicit Selections
• Remark 1: Strict Differentiability
• Definition 3: Abadie Constraint Qualification (ACQ)
• Theorem 4: QCQP Jacobian
• Remark 2: Smooth Feasible Set Assumption
• Lemma 5
• Proposition 6