Feasible Learning

TL;DR

This work reframes learning as a feasibility problem by enforcing per-sample loss bounds $g_i(\boldsymbol{\theta}) \le \epsilon$ across all training points, rather than optimizing average loss as in ERM. It develops a primal-dual optimization scheme that dynamically reweights samples via dual variables $\boldsymbol{\lambda}$, enabling scalable training while preserving FEASIBLE solutions. To handle potential infeasibility, it introduces Resilient Feasible Learning (RFL) with slack variables $\boldsymbol{u}$ and a quadratic penalty $\frac{\alpha}{2}\|\boldsymbol{u}\|^2$, establishing a strong duality-based reformulation that supports stable optimization. Empirically, FL and RFL achieve comparable average performance to ERM on CIFAR-10, UTKFace, and DPO-style tasks, while producing a more favorable tail loss distribution (lower CVaR) and offering insight into sample difficulty through multiplier values.

Abstract

We introduce Feasible Learning (FL), a sample-centric learning paradigm where models are trained by solving a feasibility problem that bounds the loss for each training sample. In contrast to the ubiquitous Empirical Risk Minimization (ERM) framework, which optimizes for average performance, FL demands satisfactory performance on every individual data point. Since any model that meets the prescribed performance threshold is a valid FL solution, the choice of optimization algorithm and its dynamics play a crucial role in shaping the properties of the resulting solutions. In particular, we study a primal-dual approach which dynamically re-weights the importance of each sample during training. To address the challenge of setting a meaningful threshold in practice, we introduce a relaxation of FL that incorporates slack variables of minimal norm. Our empirical analysis, spanning image classification, age regression, and preference optimization in large language models, demonstrates that models trained via FL can learn from data while displaying improved tail behavior compared to ERM, with only a marginal impact on average performance.

Figures (12)

• Figure 1: Fitting polynomials of varying degrees: While ERM tends to overfit with high-degree polynomials, FL still recovers smoother solutions. This occurs even though non-smooth solutions are also part of the FL solution set, highlighting the influence of the optimization algorithm on the final solution. Due to ill-conditioning, we solve both problems using a standard convex optimization solver. The data is generated by adding Gaussian noise ($\sigma=0.2$) to a cosine wave. FL's constraint value is set to one standard deviation, $\epsilon = \sigma$. See \ref{['app:details']} for details.
• Figure 2: Empirical distribution of validation per-sample DPO losses on a fine-tuned Llama3-8B. Left: The empirical Cumulative Density Function (CDF). Right: The empirical Conditional Value at Risk (CVaR). FL results in fewer samples with very high losses and a lower average loss for those samples. The CVaR represents the average loss for samples exceeding each quantile of the loss distribution.
• Figure 3: FL for a two-dimensional classification task. The marker size of each datapoint is proportional to its corresponding multiplier value at the end of training. Points near the decision boundary have large multipliers, while those farther away have near-zero ones. The contours correspond to the level curves of the predicted probabilities.
• Figure 4: $e = 0.15$, $\lambda = 0.2$
• Figure 5: $e = 0.7$, $\lambda = 0.2$

Theorems & Definitions (4)

• Proposition 1
• Proposition 2
• proof : Proof of \ref{['prop:strong_duality']}
• proof : Proof of \ref{['prop:clamped_quadratic']}