Resilient Constrained Learning

TL;DR

This paper relaxes the learning constraints in a way that contemplates how much they affect the task at hand by balancing the performance gains obtained from the relaxation against a user-defined cost of that relaxation.

Abstract

When deploying machine learning solutions, they must satisfy multiple requirements beyond accuracy, such as fairness, robustness, or safety. These requirements are imposed during training either implicitly, using penalties, or explicitly, using constrained optimization methods based on Lagrangian duality. Either way, specifying requirements is hindered by the presence of compromises and limited prior knowledge about the data. Furthermore, their impact on performance can often only be evaluated by actually solving the learning problem. This paper presents a constrained learning approach that adapts the requirements while simultaneously solving the learning task. To do so, it relaxes the learning constraints in a way that contemplates how much they affect the task at hand by balancing the performance gains obtained from the relaxation against a user-defined cost of that relaxation. We call this approach resilient constrained learning after the term used to describe ecological systems that adapt to disruptions by modifying their operation. We show conditions under which this balance can be achieved and introduce a practical algorithm to compute it, for which we derive approximation and generalization guarantees. We showcase the advantages of this resilient learning method in image classification tasks involving multiple potential invariances and in heterogeneous federated learning.

Figures (10)

• Figure 1: Resilient equilibrium from Def. \ref{['def_resilient']} for $h(u) = u^2/2$. The shaded area indicates infeasible specifications: (a) nominal specification ($u = 0$) is feasible and easy to satisfy; (b) nominal specification is feasible but difficult to satisfy (close to infeasible); (c) nominal specification is infeasible.
• Figure 2: (Left) Constraint relaxation and relative difficulty for federated learning under heterogeneous class imbalance across clients (crosses). We plot the perturbation $u_c$ against the fraction of the dataset of client $c$ from the minority class, which is associated to how difficult it is to satisfy the constraint, since minority classes typically have higher loss. (Middle) Relaxation cost parameter ($h({\mathbf u}) = \alpha \Vert {\mathbf u} \Vert_2^2$) vs. final training loss and perturbation norm. (Right) Constraint violations on train and test sets
• Figure 3: (Left) Constraint Relaxation and Relative difficulty for Synthetically invariant datasets. Bars denote the perturbation ${\mathbf u}_i$ associated with different transformations, and the x-axis shows the synthetic invariance of the dataset. (Right) Sensitivity (Dual variables) for Synthetically invariant datasets. Bars correspond to the values of dual variables (for all constraints) at the end of training. We compare the resilient and constrained approach accross different synthetic datasets.
• Figure 4: Train loss for the worst client and averaged over all clients for different values of the perturbation cost coefficient $\alpha$.
• Figure 5: Dual variables after training with respect to constraint specification $\epsilon$.

Theorems & Definitions (15)

• Definition 1: Resilient Equilibrium
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Theorem 1
• Definition 2: PACC
• Lemma 1
• Proposition 5
• proof