Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Test Performance with Importance Sampling--a Subpopulation Perspective

Hongyu Shen, Zhizhen Zhao

TL;DR

The paper tackles subpopulation shifts that degrade average test accuracy under ERM by introducing the dataset bias analysis (DBA) framework, grounded in importance sampling. It reframes learning as optimizing a weight-adjusted objective and explicitly states data-generation assumptions, enabling principled comparisons across methods. The authors propose three dataset bias correction methods (DBCM) to estimate $p(s|y,x,I_{tr})$ under varying data access, with a unified training algorithm and theoretical links showing how the test performance aligns with the weighted objective. They reinterpret existing subpopulation techniques within the DBA view, clarifying when average vs. worst-group performance is affected by objective misspecification. Empirically, DBCM achieves state-of-the-art results on ColorMNIST, Waterbirds, and CivilComments for both average and worst-group accuracy, while maintaining robustness when the spurious attribute is unknown.

Abstract

Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes. Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area. In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem. On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works. This helps to uncover the cause of the dropped average accuracy. We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different. On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem. In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases. And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets.

Boosting Test Performance with Importance Sampling--a Subpopulation Perspective

TL;DR

The paper tackles subpopulation shifts that degrade average test accuracy under ERM by introducing the dataset bias analysis (DBA) framework, grounded in importance sampling. It reframes learning as optimizing a weight-adjusted objective and explicitly states data-generation assumptions, enabling principled comparisons across methods. The authors propose three dataset bias correction methods (DBCM) to estimate under varying data access, with a unified training algorithm and theoretical links showing how the test performance aligns with the weighted objective. They reinterpret existing subpopulation techniques within the DBA view, clarifying when average vs. worst-group performance is affected by objective misspecification. Empirically, DBCM achieves state-of-the-art results on ColorMNIST, Waterbirds, and CivilComments for both average and worst-group accuracy, while maintaining robustness when the spurious attribute is unknown.

Abstract

Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes. Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area. In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem. On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works. This helps to uncover the cause of the dropped average accuracy. We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different. On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem. In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases. And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets.

Paper Structure

This paper contains 33 sections, 3 theorems, 25 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Importance Sampling
  4. Subpopulation Survey
  5. Subpopulation Method
  6. Method
  7. Dataset Bias Analysis Framework
  8. Dataset Bias Correction Method
  9. Attribute $s$ is Known
  10. Attribute $s$ is Unknown and $\mathcal{D}_\text{tr} \sim \mathcal{D}_\text{va}$
  11. Attribute $s$ is Unknown and $\mathcal{D}_\text{tr} \nsim \mathcal{D}_\text{va}$
  12. Choose Models
  13. DBA Interpretation on Existing Work
  14. Experiment
  15. Experimental Setup
  16. ...and 18 more sections

Key Result

Theorem 1

Given Assumption assume_support, assume_data_gen, assume_s_uniform, assume_proposal_1, and assume_proposal_2 hold, the optimization of Eq. eq_adjust_obj with the following weight function $g(x, y, I_\text{tr}, I_\text{te})$ directly maximizes the testing performance: where $p(y\vert m_1, I_\text{tr}) = \frac{p(y\vert I_\text{tr}) - p(m_0\vert I_\text{tr})\cdot p(y\vert I_\text{tr})}{p(m_1\vert I_

Figures (2)

  • Figure 1: An image example on subpopulation shift. The left panel contains images where digits and colors are correlated, whereas the right panel does not exhibit such correlation.
  • Figure : The universal algorithm for optimizing $q(y\vert x, I_\text{tr})$.

Theorems & Definitions (6)

  • Claim 1
  • Claim 2
  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3