Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimization in Theory and Practice

Stephen J. Wright

TL;DR

<3-5 sentence high-level summary> The paper surveys the interplay between theory and practice in continuous optimization, focusing on linear programming and unconstrained smooth optimization. It covers formulations, optimality conditions, and non-asymptotic complexity, tracing the evolution from simplex to interior-point and first-order methods, and extending to Newton- and quasi-Newton approaches as well as tractable nonconvex regimes. It highlights key complexity notions (iteration, operation, and oracle complexity), gaps between worst-case theory and practical performance, and the emergence of modern analytical frameworks (PEP, smoothed analysis) that guide algorithm design and understanding. The discussion emphasizes practical impact for large-scale problems and machine learning, where both classical results and new techniques inform solver development and performance.

Abstract

Algorithms for continuous optimization problems have a rich history of design and innovation over the past several decades, in which mathematical analysis of their convergence and complexity properties plays a central role. Besides their theoretical properties, optimization algorithms are interesting also for their practical usefulness as computational tools for solving real-world problems. There are often gaps between the practical performance of an algorithm and what can be proved about it. These two facets of the field -- the theoretical and the practical -- interact in fascinating ways, each driving innovation in the other. This work focuses on the development of algorithms in two areas -- linear programming and unconstrained minimization of smooth functions -- outlining major algorithm classes in each area along with their theoretical properties and practical performance, and highlighting how advances in theory and practice have influenced each other in these areas. In discussing theory, we focus mainly on non-asymptotic complexity, which are upper bounds on the amount of computation required by a given algorithm to find an approximate solution of problems in a given class.

Optimization in Theory and Practice

TL;DR

<3-5 sentence high-level summary> The paper surveys the interplay between theory and practice in continuous optimization, focusing on linear programming and unconstrained smooth optimization. It covers formulations, optimality conditions, and non-asymptotic complexity, tracing the evolution from simplex to interior-point and first-order methods, and extending to Newton- and quasi-Newton approaches as well as tractable nonconvex regimes. It highlights key complexity notions (iteration, operation, and oracle complexity), gaps between worst-case theory and practical performance, and the emergence of modern analytical frameworks (PEP, smoothed analysis) that guide algorithm design and understanding. The discussion emphasizes practical impact for large-scale problems and machine learning, where both classical results and new techniques inform solver development and performance.

Abstract

Algorithms for continuous optimization problems have a rich history of design and innovation over the past several decades, in which mathematical analysis of their convergence and complexity properties plays a central role. Besides their theoretical properties, optimization algorithms are interesting also for their practical usefulness as computational tools for solving real-world problems. There are often gaps between the practical performance of an algorithm and what can be proved about it. These two facets of the field -- the theoretical and the practical -- interact in fascinating ways, each driving innovation in the other. This work focuses on the development of algorithms in two areas -- linear programming and unconstrained minimization of smooth functions -- outlining major algorithm classes in each area along with their theoretical properties and practical performance, and highlighting how advances in theory and practice have influenced each other in these areas. In discussing theory, we focus mainly on non-asymptotic complexity, which are upper bounds on the amount of computation required by a given algorithm to find an approximate solution of problems in a given class.
Paper Structure (41 sections, 54 equations, 1 figure, 1 algorithm)

This paper contains 41 sections, 54 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction.
  2. Formulations and Optimality Conditions.
  3. Unconstrained Optimization.
  4. Linear Programming.
  5. Convergence and Complexity in Optimization.
  6. Iteration Complexity.
  7. Operation Complexity.
  8. Oracle Complexity.
  9. Lower Bounds and Optimal Algorithms.
  10. Gaps between Theory and Practice.
  11. Historical Perspectives.
  12. Linear Programming.
  13. Simplex Method.
  14. Average-case Analyses.
  15. Smoothed Analysis.
  16. ...and 26 more sections

Figures (1)

  • Figure 4.1: Progression of complexity bounds in matrix multiplication (blue) and linear programming (red). The final dashed piece of the red line shows the complexities of the methods of cohen2021solvingvan2020deterministic and jiang2021faster, respectively, that would apply if the exponent of matrix multiplication were shown to be $2$. (Credit: Yin-Tat Lee.)