Noise-Aware Generalization: Robustness to In-Domain Noise and Out-of-Domain Generalization

TL;DR

This paper defines Noise-Aware Generalization (NAG), addressing robustness to in-domain label noise alongside out-of-domain generalization. It proposes DL4ND, a cross-domain noise-detection mechanism that uses low-loss proxies to refine noisy labels, and demonstrates how to integrate this with various DG methods (DL4ND+DG). Through experiments on real-world VLCS/CHAMMI-CP and synthetic OfficeHome/TerraIncognita datasets, it shows DL4ND consistently improves both ID and OOD performance, with gains up to 22% in high-noise settings and competitive results against strong DG baselines. The work also analyzes pitfalls of naive LNL+DG combinations and offers practical insights for combining LNL and DG, highlighting that domain-label-informed, regularization-focused strategies and high-quality label refinement are key for robust NAG performance.

Abstract

Multi-source Domain Generalization (DG) aims to improve model robustness to new distributions. However, DG methods often overlook the effect of label noise, which can confuse a model during training, reducing performance. Limited prior work has analyzed DG method's noise-robustness, typically focused on an analysis of existing methods rather than new solutions. In this paper, we investigate this underexplored space, where models are evaluated under both distribution shifts and label noise, which we refer to as Noise-Aware Generalization (NAG). A natural solution to address label noise would be to combine a Learning with Noisy Labels (LNL) method with those from DG. Many LNL methods aim to detect distribution shifts in a class's samples, i.e., they assume that distribution shifts often correspond to label noise. However, in NAG distribution shifts can be due to label noise or domain shifts, breaking the assumptions used by LNL methods. A naive solution is to make a similar assumption made by many DG methods, where we presume to have domain labels during training, enabling us to isolate the two types of shifts. However, this ignores valuable cross-domain information. Specifically, our proposed DL4ND approach improves noise detection by taking advantage of the observation that noisy samples that may appear indistinguishable within a single domain often show greater variation when compared across domains. Experiments show that DL4ND significantly improves performance across four diverse datasets, offering a promising direction for tackling NAG.

Figures (7)

• Figure 1: NAG Task Challenge. Noisy label samples and those from other distributions can be both similar and dissimilar to the true class, complicating the task of generalizing. While prior work only evaluates robustness to these distributions (e.g., qiao2024understandingseo2020learning), our paper takes a step toward addressing these challenges directly.
• Figure 2: The relationship between our task and related works. DG typically methods either ignore in-domain performance (e.g., sagawa2019distributionallyrame2022fishrkrueger2021outzhang2024domainwortsman2022robust), label noise (e.g., teterwak2023erm++cha2022mirocha2021swadwang2023sharpness), or both (e.g., teterwak2023erm++cha2022mirocha2021swadwang2023sharpness). Analogously, LNL methods may report in-domain performance and are robust to label noise, but ignore domain shifts liu2020earlyli2023disckarim2022uniconzhao2024estimating. NAG explores methods that are effect in all three aspects, making for more robust models.
• Figure 3: Box plot of distance distributions across classes and domains. The distance is measured between each sample and its (class, domain) group average. (a) The group average is calculated from all training samples. The red box highlights overlapping distributions, indicating the challenge of distinguishing samples with class and domain shift. (b) The group average is calculated from low-loss samples, showing no overlapping distributions.
• Figure 4: Real-world datasets with in-domain noise and multi-domain distribution. VLCS (web/user data) fang2013unbiased, and CHAMMI-CP (biomedical images) chen2024chammi. VLCS faces label noise from poor annotations and domain shifts from varying data sources, while CHAMMI-CP deals with ambiguous features and varying experimental environments.
• Figure 5: A solution to the NAG challenge — DL4ND. Comparing samples across different domains helps avoid spurious similar features within the current domain and enables decisions based on invariant intrinsic features.

Theorems & Definitions (1)

• Theorem 1