Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalizing to unseen domains via distribution matching

Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, Ioannis Mitliagkas

TL;DR

The paper tackles domain generalization by proving that small pairwise divergences among observed domains imply small divergences among mixtures within their convex hull, enabling generalization to unseen domains. It introduces G2DM, an adversarial distribution-matching method that minimizes pairwise source-domain discrepancies using one-vs-all discriminators and stabilizes training with random projections. Theoretical bounds on unseen-domain risk are provided, with corollaries under covariate shift, and practical algorithms are designed to optimize the bound using only source data. Empirically, G2DM improves performance over ERM and competitive domain-adaptation methods on DG benchmarks (VLCS, PACS) and EEG data, demonstrating effective generalization without access to test-domain data and highlighting the value of diverse training domains.

Abstract

Supervised learning results typically rely on assumptions of i.i.d. data. Unfortunately, those assumptions are commonly violated in practice. In this work, we tackle such problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions). Our work relies on the following lemma: by minimizing a notion of discrepancy between all pairs from a set of given domains, we also minimize the discrepancy between any pairs of mixtures of domains. Using this result, we derive a generalization bound for our setting. We then show that low risk over unseen domains can be achieved by representing the data in a space where (i) the training distributions are indistinguishable, and (ii) relevant information for the task at hand is preserved. Minimizing the terms in our bound yields an adversarial formulation which estimates and minimizes pairwise discrepancies. We validate our proposed strategy on standard domain generalization benchmarks, outperforming a number of recently introduced methods. Notably, we tackle a real-world application where the underlying data corresponds to multi-channel electroencephalography time series from different subjects, each considered as a distinct domain.

Generalizing to unseen domains via distribution matching

TL;DR

The paper tackles domain generalization by proving that small pairwise divergences among observed domains imply small divergences among mixtures within their convex hull, enabling generalization to unseen domains. It introduces G2DM, an adversarial distribution-matching method that minimizes pairwise source-domain discrepancies using one-vs-all discriminators and stabilizes training with random projections. Theoretical bounds on unseen-domain risk are provided, with corollaries under covariate shift, and practical algorithms are designed to optimize the bound using only source data. Empirically, G2DM improves performance over ERM and competitive domain-adaptation methods on DG benchmarks (VLCS, PACS) and EEG data, demonstrating effective generalization without access to test-domain data and highlighting the value of diverse training domains.

Abstract

Supervised learning results typically rely on assumptions of i.i.d. data. Unfortunately, those assumptions are commonly violated in practice. In this work, we tackle such problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions). Our work relies on the following lemma: by minimizing a notion of discrepancy between all pairs from a set of given domains, we also minimize the discrepancy between any pairs of mixtures of domains. Using this result, we derive a generalization bound for our setting. We then show that low risk over unseen domains can be achieved by representing the data in a space where (i) the training distributions are indistinguishable, and (ii) relevant information for the task at hand is preserved. Minimizing the terms in our bound yields an adversarial formulation which estimates and minimizes pairwise discrepancies. We validate our proposed strategy on standard domain generalization benchmarks, outperforming a number of recently introduced methods. Notably, we tackle a real-world application where the underlying data corresponds to multi-channel electroencephalography time series from different subjects, each considered as a distinct domain.

Paper Structure

This paper contains 29 sections, 23 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. Related work
  5. Generalization guarantees for domain adaptation
  6. Learning domain agnostic representations for domain generalization
  7. Formalizing domain generalization
  8. Matching distributions in the convex hull
  9. Generalizing to unseen domains
  10. Practical contributions
  11. Experimental Setup and Results
  12. Evaluation under covariate shift
  13. Checking $\boldsymbol{\mathcal{H}}$-divergences across sources and unseen domains
  14. The effect of different access methods to test data during training
  15. Evaluation beyond the covariate shift assumption
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Differences between estimated pairwise $\mathcal{H}$-divergences under ERM and G2DM on PACS (captions denote unseen domains). Higher values indicate that G2DM better matched domains. Overall, G2DM is able to decrease pairwise discrepancies.
  • Figure 2: Proposed approach illustration.
  • Figure 3: Accuracy obtained on the PACS benchmark using Sketch as target domain.