Code generation for solving and differentiating through convex optimization problems

TL;DR

The paper tackles the problem of differentiating through parametrized convex optimization problems by generating specialized C code for the parameter-to-solution map using CVXPYgen, built on CVXPY. It combines disciplined parametrized programming with KKT-based differentiation of LPs/QPs and employs low-rank LDL updates to enable fast gradient computations. The authors demonstrate substantial speedups (roughly an order of magnitude in many cases) and validate the approach through three system-tuning and application-domain experiments, including elastic net hyper-parameter tuning, an ADP controller, and portfolio optimization. This work enables efficient, embedded-ready gradient evaluation for optimize-then-tine workflows, with practical impact on machine learning hyper-parameter tuning, control policy design, and financial back-testing.

Abstract

We introduce custom code generation for parametrized convex optimization problems that supports evaluating the derivative of the solution with respect to the parameters, i.e., differentiating through the optimization problem. We extend the open source code generator CVXPYgen, which itself extends CVXPY, a Python-embedded domain-specific language with a natural syntax for specifying convex optimization problems, following their mathematical description. Our extension of CVXPYgen adds a custom C implementation to differentiate the solution of a convex optimization problem with respect to its parameters, together with a Python wrapper for prototyping and desktop (non-embedded) applications. We give three representative application examples: Tuning hyper-parameters in machine learning; choosing the parameters in an approximate dynamic programming (ADP) controller; and adjusting the parameters in an optimization based financial trading engine via back-testing, i.e., simulation on historical data. While differentiating through convex optimization problems is not new, CVXPYgen is the first tool that generates custom C code for the task, and increases the computation speed by about an order of magnitude in most applications, compared to CVXPYlayers, a general-purpose tool for differentiating through convex optimization problems.

Figures (7)

• Figure 1: Comparison of convex optimization problem parsing and solving/differentiating approaches.
• Figure 2: Code generation and CVXPYlayers interface for elastic net example. The integers m, n, and J, the constant y, the function Gamma, and the torch tensors w, mu, and nu are pre-defined.
• Figure 3: CV loss over tuning iterations.
• Figure 4: Thresholds before (magenta) and after tuning (blue). The dashed black lines show the range of possible winsorization thresholds.
• Figure 5: Control performance over tuning iterations.