Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AL-CoLe: Augmented Lagrangian for Constrained Learning

Ignacio Boero, Ignacio Hounie, Alejandro Ribeiro

TL;DR

This work studies constrained learning in non-convex models through Augmented Lagrangian (AL) methods. It proves strong duality between the primal and augmented dual problems under mild assumptions and shows that empirical AL retains these properties, enabling PAC-Constrained (PACC) learnability with provable bounds on suboptimality and infeasibility. The authors develop an augmented dual ascent algorithm with an increased-shifted penalty scheme that guarantees primal iterates converge to feasible and near-optimal solutions without needing iterate averaging. Empirically, on fairness-constrained classification tasks, AL demonstrates smoother dual dynamics, improved constraint satisfaction, and better last-iterate performance than standard dual ascent, highlighting practical benefits for safe and fair learning.

Abstract

Despite the non-convexity of most modern machine learning parameterizations, Lagrangian duality has become a popular tool for addressing constrained learning problems. We revisit Augmented Lagrangian methods, which aim to mitigate the duality gap in non-convex settings while requiring only minimal modifications, and have remained comparably unexplored in constrained learning settings. We establish strong duality results under mild conditions, prove convergence of dual ascent algorithms to feasible and optimal primal solutions, and provide PAC-style generalization guarantees. Finally, we demonstrate its effectiveness on fairness constrained classification tasks.

AL-CoLe: Augmented Lagrangian for Constrained Learning

TL;DR

This work studies constrained learning in non-convex models through Augmented Lagrangian (AL) methods. It proves strong duality between the primal and augmented dual problems under mild assumptions and shows that empirical AL retains these properties, enabling PAC-Constrained (PACC) learnability with provable bounds on suboptimality and infeasibility. The authors develop an augmented dual ascent algorithm with an increased-shifted penalty scheme that guarantees primal iterates converge to feasible and near-optimal solutions without needing iterate averaging. Empirically, on fairness-constrained classification tasks, AL demonstrates smoother dual dynamics, improved constraint satisfaction, and better last-iterate performance than standard dual ascent, highlighting practical benefits for safe and fair learning.

Abstract

Despite the non-convexity of most modern machine learning parameterizations, Lagrangian duality has become a popular tool for addressing constrained learning problems. We revisit Augmented Lagrangian methods, which aim to mitigate the duality gap in non-convex settings while requiring only minimal modifications, and have remained comparably unexplored in constrained learning settings. We establish strong duality results under mild conditions, prove convergence of dual ascent algorithms to feasible and optimal primal solutions, and provide PAC-style generalization guarantees. Finally, we demonstrate its effectiveness on fairness constrained classification tasks.
Paper Structure (13 sections, 9 theorems, 38 equations, 3 figures)

This paper contains 13 sections, 9 theorems, 38 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Augmented Lagrangian for constrained learning
  3. Statistical Constrained Learning
  4. Augmented Lagrangian and Duality
  5. Empirical Dual Learning
  6. Augmented dual ascent
  7. Numerical results
  8. Conclusions
  9. Duality of the Augmented Lagrangian
  10. Properties of the dual problem
  11. Strong duality
  12. Optimal Dual Solutions
  13. PACC-Learnability

Key Result

Theorem 2.1

If Assumptions ass:l0_bounded--ass:feasible_compact hold, then strong duality holds for problem eq:constrained_risk, i.e.,

Figures (3)

  • Figure 1: Dynamics of lagrange multipliers $\lambda$ training iterations, for both the augmented Lagrangian and regular Lagrangian methods. Both methods use the same hyperparameters, including the dual learning rate.
  • Figure 2: (Left): Evolution of constraint slacks $s_i^t$ during training for easier constraints. (Right): Same for harder constraints. A constraint is satisfied when $s_i^t<0$. We compare augmented and standard Lagrangian methods.
  • Figure 3: (Left): Classification accuracy over held-out data for unconstrained and constrained methods. (Right): Counterfactual fairness measured as the fraction of predictions that do not change after modifying a sensitive attribute.

Theorems & Definitions (21)

  • Theorem 2.1: Strong Duality
  • proof
  • Theorem 2.2: Optimal Dual Solutions
  • proof
  • Theorem 2.3: PAC-Constrained Dual Learning
  • proof
  • Theorem 3.1: Primal Convergence
  • proof
  • Proposition A.1: Weak Duality
  • proof
  • ...and 11 more