Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributionally Robust Optimization via Generative Ambiguity Modeling

Jiaqi Wen, Jianyi Yang

TL;DR

The paper tackles the challenge of constructing ambiguity sets for Distributionally Robust Optimization that are both expressive and tractable. It introduces Generative Ambiguity Sets (GAS) and a practical GAS-DRO algorithm that optimizes over a parameterized space of likelihood-based generative models (e.g., diffusion models and VAEs) while constraining reconstruction loss via $J(\theta,P_0)\le \epsilon$, ensuring consistency with the nominal distribution $P_0$. The inner maximization is tackled with dual learning and policy optimization (PPO), enabling tractable optimization in a finite parameter space, and the authors prove stationary convergence of GAS-DRO along with bounds on the inner-max error and a KL-divergence constraint. Empirically, GAS-DRO demonstrates superior Out-of-Distribution generalization on time-series and image classification tasks, outperforming Wasserstein- and KL-based DRO methods and prior generative-DRO approaches. The work offers a principled, scalable DRO framework that leverages the representational power of generative models to balance realism and robustness in distribution shifts.

Abstract

This paper studies Distributionally Robust Optimization (DRO), a fundamental framework for enhancing the robustness and generalization of statistical learning and optimization. An effective ambiguity set for DRO must involve distributions that remain consistent to the nominal distribution while being diverse enough to account for a variety of potential scenarios. Moreover, it should lead to tractable DRO solutions. To this end, we propose generative model-based ambiguity sets that capture various adversarial distributions beyond the nominal support space while maintaining consistency with the nominal distribution. Building on this generative ambiguity modeling, we propose DRO with Generative Ambiguity Set (GAS-DRO), a tractable DRO algorithm that solves the inner maximization over the parameterized generative model space. We formally establish the stationary convergence performance of GAS-DRO. We implement GAS-DRO with a diffusion model and empirically demonstrate its superior Out-of-Distribution (OOD) generalization performance in ML tasks.

Distributionally Robust Optimization via Generative Ambiguity Modeling

TL;DR

The paper tackles the challenge of constructing ambiguity sets for Distributionally Robust Optimization that are both expressive and tractable. It introduces Generative Ambiguity Sets (GAS) and a practical GAS-DRO algorithm that optimizes over a parameterized space of likelihood-based generative models (e.g., diffusion models and VAEs) while constraining reconstruction loss via , ensuring consistency with the nominal distribution . The inner maximization is tackled with dual learning and policy optimization (PPO), enabling tractable optimization in a finite parameter space, and the authors prove stationary convergence of GAS-DRO along with bounds on the inner-max error and a KL-divergence constraint. Empirically, GAS-DRO demonstrates superior Out-of-Distribution generalization on time-series and image classification tasks, outperforming Wasserstein- and KL-based DRO methods and prior generative-DRO approaches. The work offers a principled, scalable DRO framework that leverages the representational power of generative models to balance realism and robustness in distribution shifts.

Abstract

This paper studies Distributionally Robust Optimization (DRO), a fundamental framework for enhancing the robustness and generalization of statistical learning and optimization. An effective ambiguity set for DRO must involve distributions that remain consistent to the nominal distribution while being diverse enough to account for a variety of potential scenarios. Moreover, it should lead to tractable DRO solutions. To this end, we propose generative model-based ambiguity sets that capture various adversarial distributions beyond the nominal support space while maintaining consistency with the nominal distribution. Building on this generative ambiguity modeling, we propose DRO with Generative Ambiguity Set (GAS-DRO), a tractable DRO algorithm that solves the inner maximization over the parameterized generative model space. We formally establish the stationary convergence performance of GAS-DRO. We implement GAS-DRO with a diffusion model and empirically demonstrate its superior Out-of-Distribution (OOD) generalization performance in ML tasks.
Paper Structure (46 sections, 7 theorems, 90 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 46 sections, 7 theorems, 90 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Ambiguity Modeling in DRO
  4. Ambiguity Modeling based on Generative Models
  5. Probability Modeling via Generative Models
  6. Generative Ambiguity Set
  7. Algorithm for DRO with Generative Ambiguity Set
  8. Analysis
  9. Experiments
  10. Conclusion
  11. Details of GAS-DRO (Algorithm \ref{['DAG-DRL']}) with Diffusion Models
  12. Objective Derivation by Policy Gradient
  13. Objective Derivation by Proximal Policy Optimization
  14. Details of GAS-DRO (Algorithm \ref{['DAG-DRL']}) with VAEs
  15. Ambiguity set For VAE
  16. ...and 31 more sections

Key Result

Lemma 1

The inclusive KL-divergence between the nominal distribution $P_0$ and the sampling distribution $P_{\theta}$ of a likelihood-based generative model can be bounded as where $J(\theta,P_0)$ is the reconstruction loss which can be $J_{\mathrm{DM}}(\theta,P_0)$ in equation eqn:diffusion_loss or $J_{\mathrm{VAE}}(\theta,P_0)$ in the first term of equation eqm:elbo, $R(p', P_0)$ is a prior matching lo

Figures (5)

  • Figure 1: DRO with generative ambiguity set
  • Figure 2: Gaussian perturbation strength
  • Figure 3: Perlin perturbation strength
  • Figure 4: Cutout perturbation strength
  • Figure 5: Effect of budget $\epsilon$ in GAS-DRO

Theorems & Definitions (16)

  • Lemma 1
  • Theorem 1: Convergence of Inner Maximization
  • Theorem 2: Convergence of GAS-DRO
  • Definition 1: $\Gamma-$expressivity of the generative model
  • proof
  • Lemma 2
  • proof
  • proof
  • proof
  • Corollary 1
  • ...and 6 more