A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization

TL;DR

This work tackles optimization for compositional entropic risk minimization, where each data point incurs a loss of Log-E-Exp form. It introduces SCENT, a geometry-aware stochastic algorithm that optimizes a min-min dual formulation via a stochastic proximal mirror-descent update on the dual variable, using a Bregman divergence induced by e^{-\nu} to handle large smoothness constants. The authors establish a convex-case O(1/\sqrt{T}) convergence rate, analyze the intrinsic complexity through a second-order moment ratio kappa, and show SCENT's dual updates provide practical advantages over standard SGD. Empirically, SCENT yields consistent improvements on extreme classification, partial AUC maximization, CLIP, and distributionally robust optimization tasks, validating its theoretical guarantees and scalability to massive problems.

Abstract

This paper studies optimization for a family of problems termed $\textbf{compositional entropic risk minimization}$, in which each data's loss is formulated as a Log-Expectation-Exponential (Log-E-Exp) function. The Log-E-Exp formulation serves as an abstraction of the Log-Sum-Exponential (LogSumExp) function when the explicit summation inside the logarithm is taken over a gigantic number of items and is therefore expensive to evaluate. While entropic risk objectives of this form arise in many machine learning problems, existing optimization algorithms suffer from several fundamental limitations including non-convergence, numerical instability, and slow convergence rates. To address these limitations, we propose a geometry-aware stochastic algorithm, termed $\textbf{SCENT}$, for the dual formulation of entropic risk minimization cast as a min--min optimization problem. The key to our design is a $\textbf{stochastic proximal mirror descent (SPMD)}$ update for the dual variable, equipped with a Bregman divergence induced by a negative exponential function that faithfully captures the geometry of the objective. Our main contributions are threefold: (i) we establish an $O(1/\sqrt{T})$ convergence rate of the proposed SCENT algorithm for convex problems; (ii) we theoretically characterize the advantages of SPMD over standard SGD update for optimizing the dual variable; and (iii) we demonstrate the empirical effectiveness of SCENT on extreme classification, partial AUC maximization, contrastive learning and distributionally robust optimization, where it consistently outperforms existing baselines.

Figures (6)

• Figure 1: Ratio between the error of SPMD and that of SGD when trained on Gaussian noise with different means and variances.
• Figure 2: (\ref{['fig:baselines:glint']}): Cross-entropy loss curves of different methods on the training set (left) and validation set (right) of Glint360K. (\ref{['fig:baselines:treeoflife']}): Cross-entropy loss curves of different methods on the training set (left) and validation dataset (right) of TreeOfLife-10M.
• Figure 4: (\ref{['fig:baselines:glint']}): cross-entropy loss of different methods on the training set (left) and validation dataset (right) of Glint360K, using SGD with momentum optimizer. (\ref{['fig:baselines:treeoflife']}): cross-entropy loss of different methods on the training set (left) and validation dataset (right) of TreeOfLife-10M, using SGD with momentum optimizer.
• Figure 5: Training loss curves of different methods using SGD with momentum optimizer for partial AUC maximization. (\ref{['fig:cifar10_pauc_momentum']}): on the dataset CIFAR-10 with $\tau=0.05$ (left) and $\tau=0.1$. (\ref{['fig:CIFAR100_pauc_momentum']}): on the dataset CIFAR-100 with $\tau=0.05$ (left) and $\tau=0.1$.
• Figure 6: Error between $\nu_{t}$ and $\nu_*$ when trained using different methods on Gaussian noise with different mean (top to bottom: $\mu= -1.0, -10.0$) and standard deviation (left to right: $\sigma= 0.1, 0.3, 1.0$)

Theorems & Definitions (56)

• Lemma 3.1
• Lemma 3.3
• Lemma 3.4
• Lemma 3.5
• Theorem 3.6
• Lemma 4.2
• Theorem 4.3
• Lemma 4.4
• Theorem 4.5
• Lemma 2.1