Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Optimize by Differentiable Programming

Liping Tao, Xindi Tong, Chee Wei Tan

TL;DR

The paper addresses the challenge of solving massive-scale optimization problems by combining differentiable programming with duality theory and first-order methods. It articulates how to design trainable optimization blocks within differentiable frameworks, leveraging Fenchel and Lagrangian duality to obtain dual certificates and guide learning. The work surveys theoretical foundations, software tools, and concrete strategies for embedding cone-programming solvers (via penalty, transformation, or dual reformulations) into differentiable pipelines, with PyTorch implementations for NNLS and large-scale distributed settings. Through case studies on LP, OPF, Laplacian regularization, and neural network verification, it demonstrates end-to-end differentiable optimization that yields scalable, robust, and certifiable solutions. The results highlight a practical pathway to integrate optimization layers into learning systems, enabling adaptive, efficient, and verifiable performance on complex, high-dimensional problems.

Abstract

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as ADMM and PDHG can be learned and adapted. Case studies across LP, OPF, Laplacian regularization, and neural network verification illustrate these gains.

Learning to Optimize by Differentiable Programming

TL;DR

The paper addresses the challenge of solving massive-scale optimization problems by combining differentiable programming with duality theory and first-order methods. It articulates how to design trainable optimization blocks within differentiable frameworks, leveraging Fenchel and Lagrangian duality to obtain dual certificates and guide learning. The work surveys theoretical foundations, software tools, and concrete strategies for embedding cone-programming solvers (via penalty, transformation, or dual reformulations) into differentiable pipelines, with PyTorch implementations for NNLS and large-scale distributed settings. Through case studies on LP, OPF, Laplacian regularization, and neural network verification, it demonstrates end-to-end differentiable optimization that yields scalable, robust, and certifiable solutions. The results highlight a practical pathway to integrate optimization layers into learning systems, enabling adaptive, efficient, and verifiable performance on complex, high-dimensional problems.

Abstract

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as ADMM and PDHG can be learned and adapted. Case studies across LP, OPF, Laplacian regularization, and neural network verification illustrate these gains.
Paper Structure (39 sections, 98 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 39 sections, 98 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Differentiable Programming
  3. Theoretical Foundations of Differentiable Programming
  4. Gradient Calculus
  5. Computational Graph
  6. Automatic Differentiation
  7. Software and Frameworks of Differentiable Programming
  8. PyTorch
  9. JAX
  10. Julia
  11. Recent Implementations of Differentiable Programming in Optimization
  12. CVXPYLayer
  13. PyEPO
  14. DDNs
  15. Elements of Differentiable Programming
  16. ...and 24 more sections

Figures (7)

  • Figure 1: Learning to Optimize via Differentiable Programming by Combining Duality and First-order Methods.
  • Figure 2: Backpropagation in PyTorch via Chain Rule.
  • Figure 3: Convergence comparison of PDG, ADMM, and PDHG on the NNLS problem.
  • Figure 4: ADMM with PyTorch.
  • Figure 5: Learning with PyTorch.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Example 2.1: Backpropagation via Chain Rule and Lagrangian
  • Remark 1
  • Example 3.1: Lagrangian Duality of NNLS
  • Example 3.2: Fenchel Duality of NNLS
  • Example 3.3: Classical NNLS
  • Example 3.4: Large-Scale Distributed NNLS