Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OptNet: Differentiable Optimization as a Layer in Neural Networks

Brandon Amos, J. Zico Kolter

TL;DR

OptNet introduces differentiable optimization layers by embedding quadratic programs into neural networks, enabling exact differentiation through KKT conditions and end-to-end training of constraint-aware layers. A GPU-accelerated batched primal-dual interior point solver provides fast forward solves and yields backward gradients with negligible extra cost, making constrained inference practical within deep nets. The framework is demonstrated on tasks like total variation denoising and learning the hard constraints of Sudoku, showing improved constraint handling and generalization over traditional networks. The work broadens neural network expressiveness by integrating structured optimization, with implications for more faithful modeling of systems governed by hard constraints and dependencies. Overall, OptNet provides a principled, scalable approach to differentiable optimization within deep architectures, enabling richer representations and end-to-end learning of constrained inference problems.

Abstract

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. We explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, the method is learns to play mini-Sudoku (4x4) given just input and output games, with no a-priori information about the rules of the game; this highlights the ability of OptNet to learn hard constraints better than other neural architectures.

OptNet: Differentiable Optimization as a Layer in Neural Networks

TL;DR

OptNet introduces differentiable optimization layers by embedding quadratic programs into neural networks, enabling exact differentiation through KKT conditions and end-to-end training of constraint-aware layers. A GPU-accelerated batched primal-dual interior point solver provides fast forward solves and yields backward gradients with negligible extra cost, making constrained inference practical within deep nets. The framework is demonstrated on tasks like total variation denoising and learning the hard constraints of Sudoku, showing improved constraint handling and generalization over traditional networks. The work broadens neural network expressiveness by integrating structured optimization, with implications for more faithful modeling of systems governed by hard constraints and dependencies. Overall, OptNet provides a principled, scalable approach to differentiable optimization within deep architectures, enabling richer representations and end-to-end learning of constrained inference problems.

Abstract

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. We explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, the method is learns to play mini-Sudoku (4x4) given just input and output games, with no a-priori information about the rules of the game; this highlights the ability of OptNet to learn hard constraints better than other neural architectures.

Paper Structure

This paper contains 27 sections, 3 theorems, 26 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Energy-based learning methods
  4. Analytically
  5. Unrolling
  6. Argmin differentiation
  7. OptNet: solving optimization within a neural network
  8. An efficient batched QP solver
  9. Efficiently computing gradients
  10. Properties and representational power
  11. Limitations of the method
  12. Experimental results
  13. Batch QP solver performance
  14. Total variation denoising
  15. Baseline: Total variation denoising
  16. ...and 12 more sections

Key Result

Theorem 1

Let $z^\star(\theta)$ be the output of an OptNet layer, where $\theta = \{Q,p,A,b,G,h\}$. Assuming $Q \succ 0$ and that $A$ has full row rank, then $z^\star(\theta)$ is subdifferentiable everywhere: $\partial z^\star(\theta) \neq \varnothing$, where $\partial z^\star(\theta)$ denotes the Clarke gene

Figures (9)

  • Figure 1: Runtime of a linear layer and a QP layer, batch of 128.
  • Figure 2: Performance of Gurobi and our QP solver.
  • Figure 3: Initial and learned difference operators for denoising.
  • Figure 4: Example mini-Sudoku initial problem and solution.
  • Figure 5: Sudoku training plots.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof
  • proof
  • proof