Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Limitations of General Purpose Domain Generalisation Methods

Henry Gouk, Ondrej Bohdal, Da Li, Timothy Hospedales

TL;DR

This work establishes fundamental limits for Domain Generalisation by deriving upper bounds on the ERM excess risk and minimax lower bounds across covariate shift, bounded IPM, and bounded density ratio DG settings. It shows that, up to constants, no general-purpose DG method can substantially outperform ERM, while offering actionable guidance to optimise ERM via regularisation and invariant or well-specified hypothesis classes. The results help explain why empirically proposed DG methods often fail to beat ERM and suggest that improvements hinge on stronger, problem-specific assumptions or data-driven design of invariant representations. The analysis is complemented by experiments illustrating how carefully chosen invariances can reduce the number of training domains required for good DG performance.

Abstract

We investigate the fundamental performance limitations of learning algorithms in several Domain Generalisation (DG) settings. Motivated by the difficulty with which previously proposed methods have in reliably outperforming Empirical Risk Minimisation (ERM), we derive upper bounds on the excess risk of ERM, and lower bounds on the minimax excess risk. Our findings show that in all the DG settings we consider, it is not possible to significantly outperform ERM. Our conclusions are limited not only to the standard covariate shift setting, but also two other settings with additional restrictions on how domains can differ. The first constrains all domains to have a non-trivial bound on pairwise distances, as measured by a broad class of integral probability metrics. The second alternate setting considers a restricted class of DG problems where all domains have the same underlying support. Our analysis also suggests how different strategies can be used to optimise the performance of ERM in each of these DG setting. We also experimentally explore hypotheses suggested by our theoretical analysis.

On the Limitations of General Purpose Domain Generalisation Methods

TL;DR

This work establishes fundamental limits for Domain Generalisation by deriving upper bounds on the ERM excess risk and minimax lower bounds across covariate shift, bounded IPM, and bounded density ratio DG settings. It shows that, up to constants, no general-purpose DG method can substantially outperform ERM, while offering actionable guidance to optimise ERM via regularisation and invariant or well-specified hypothesis classes. The results help explain why empirically proposed DG methods often fail to beat ERM and suggest that improvements hinge on stronger, problem-specific assumptions or data-driven design of invariant representations. The analysis is complemented by experiments illustrating how carefully chosen invariances can reduce the number of training domains required for good DG performance.

Abstract

We investigate the fundamental performance limitations of learning algorithms in several Domain Generalisation (DG) settings. Motivated by the difficulty with which previously proposed methods have in reliably outperforming Empirical Risk Minimisation (ERM), we derive upper bounds on the excess risk of ERM, and lower bounds on the minimax excess risk. Our findings show that in all the DG settings we consider, it is not possible to significantly outperform ERM. Our conclusions are limited not only to the standard covariate shift setting, but also two other settings with additional restrictions on how domains can differ. The first constrains all domains to have a non-trivial bound on pairwise distances, as measured by a broad class of integral probability metrics. The second alternate setting considers a restricted class of DG problems where all domains have the same underlying support. Our analysis also suggests how different strategies can be used to optimise the performance of ERM in each of these DG setting. We also experimentally explore hypotheses suggested by our theoretical analysis.
Paper Structure (21 sections, 18 theorems, 72 equations, 2 figures)

This paper contains 21 sections, 18 theorems, 72 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Domain Generalisation
  4. Standard i.i.d. learning
  5. From i.i.d. to DG learning
  6. How Much Can One Possibly Improve Over ERM?
  7. The Covariate Shift Setting
  8. The Excess Risk of ERM
  9. A Lower Bound on the Minimax Excess Risk
  10. The Bounded Integral Probability Metric Setting
  11. When can $\sigma$ be small?
  12. The Bounded Density Ratio Setting
  13. Conclusion
  14. Proof of Theorem \ref{['thm:csuniform']}
  15. Proof of Theorem \ref{['thm:csexcessrisk']}
  16. ...and 6 more sections

Key Result

Theorem 1

Suppose $\ell$ takes values in $[0, 1]$ and $S_m$ is contains $m$ i.i.d. samples from $P$. The worst-case difference between the population risk and empirical risk for models selected from ${\mathcal{F}}$ is bounded, in expectation, by and with probability at least $1-\delta$ over realisations of $S_m$, we have for all $f \in {\mathcal{F}}$ that where we have used $\ell \circ {\mathcal{F}} = \{(

Figures (2)

  • Figure 1: A plate diagram describing an overview of the Domain Generalisation data generation process. The environment---a distribution over domains---is represented by $E$. During the training phase, $n$ domains, $P_1$ to $P_n$, are sampled from $E$. From each of these domains we sample $m$ training points, leading to a total of $mn$ training points. At test time, the model will encounter an indeterminate number of domains, so we do not index the $P$ nodes. From each of these domains, unlabelled points are sampled and the model is required to produce an estimate of the corresponding label.
  • Figure 2: The CNN model includes a suitable invariance that helps it generalize across domains better than the MLP model.

Theorems & Definitions (27)

  • Theorem 1: mohri2018foundations
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Corollary 1
  • Theorem 6
  • Theorem 7
  • Theorem 8
  • Theorem 8
  • ...and 17 more