Group Distributionally Robust Optimization with Flexible Sample Queries

TL;DR

This work extends group distributionally robust optimization (GDRO) by introducing flexible, time-varying sample queries. By recasting GDRO as a two-player game between a w-player (non-oblivious online convex optimization) and a q-player (non-oblivious prediction with limited advice), the authors develop novel PLA algorithms and integrate them with Follow-The-Regularized-Leader updates to achieve anytime operation. They establish high-probability, time-uniform regret and optimization error bounds that scale as $O\left(\frac{1}{t}\sqrt{\sum_{j=1}^t \frac{m}{r_j}\log m}\right)$, with fixed-sample-size consequences $O\left(\frac{m\log m}{\epsilon^2}\right)$ for $r\in[m]$ and improvements over prior bounds for $r=1$ or $m$. Empirical results on synthetic and real multi-class data validate faster convergence under dynamic sampling and demonstrate the practical viability of the proposed flexible-sampling GDRO framework.

Abstract

Group distributionally robust optimization (GDRO) aims to develop models that perform well across $m$ distributions simultaneously. Existing GDRO algorithms can only process a fixed number of samples per iteration, either 1 or $m$, and therefore can not support scenarios where the sample size varies dynamically. To address this limitation, we investigate GDRO with flexible sample queries and cast it as a two-player game: one player solves an online convex optimization problem, while the other tackles a prediction with limited advice (PLA) problem. Within such a game, we propose a novel PLA algorithm, constructing appropriate loss estimators for cases where the sample size is either 1 or not, and updating the decision using follow-the-regularized-leader. Then, we establish the first high-probability regret bound for non-oblivious PLA. Building upon the above approach, we develop a GDRO algorithm that allows an arbitrary and varying sample size per round, achieving a high-probability optimization error bound of $O\left(\frac{1}{t}\sqrt{\sum_{j=1}^t \frac{m}{r_j}\log m}\right)$, where $r_t$ denotes the sample size at round $t$. This result demonstrates that the optimization error decreases as the number of samples increases and implies a consistent sample complexity of $O(m\log (m)/ε^2)$ for any fixed sample size $r\in[m]$, aligning with existing bounds for cases of $r=1$ or $m$. We validate our approach on synthetic binary and real-world multi-class datasets.

Figures (7)

• Figure 1: Illustration of GDRO with fixed sample sizes ($1$, left; $m$, middle) and flexible sample queries (right). Squares with different colors represent distinct distributions, with those that are ticked indicating the selected distributions queried for samples, and the others representing the unselected distributions.
• Figure 2: GDRO under varying sample sizes: max risk versus the number of iterations
• Figure 3: GDRO under varying sample sizes: max risk versus the running time
• Figure 4: HYB under different fixed sample sizes: max risk versus the number of iterations
• Figure 5: HYB under different fixed sample sizes: max risk versus the number of samples

Theorems & Definitions (29)

• Lemma 1
• Lemma 2
• Remark 1
• Lemma 3
• Remark 2
• Theorem 1
• Remark 3
• Remark 4
• Theorem 2
• Remark 5