Why Can Accurate Models Be Learned from Inaccurate Annotations?

TL;DR

This work studies why accurate models can be learned from inaccurate annotations by examining how label noise reshapes weight matrices. It shows empirically that moderate label inaccuracy mainly distorts lower singular components while leaving the principal subspace largely intact, and provides a theoretical bound via perturbation analysis and the Davis–Kahan theorem. The authors formalize this insight into a lightweight plug-in, LIP, consisting of Principles Subspace Preservation and Label Ambiguity Purification, which is training-free and can be added to existing methods to boost performance under noisy or partial labels. Extensive experiments across diverse datasets demonstrate consistent gains for a broad range of baselines, under asymmetric and symmetric noise, highlighting both the theoretical significance and practical utility for robust weakly supervised learning.

Abstract

Learning from inaccurate annotations has gained significant attention due to the high cost of precise labeling. However, despite the presence of erroneous labels, models trained on noisy data often retain the ability to make accurate predictions. This intriguing phenomenon raises a fundamental yet largely unexplored question: why models can still extract correct label information from inaccurate annotations remains unexplored. In this paper, we conduct a comprehensive investigation into this issue. By analyzing weight matrices from both empirical and theoretical perspectives, we find that label inaccuracy primarily accumulates noise in lower singular components and subtly perturbs the principal subspace. Within a certain range, the principal subspaces of weights trained on inaccurate labels remain largely aligned with those learned from clean labels, preserving essential task-relevant information. We formally prove that the angles of principal subspaces exhibit minimal deviation under moderate label inaccuracy, explaining why models can still generalize effectively. Building on these insights, we propose LIP, a lightweight plug-in designed to help classifiers retain principal subspace information while mitigating noise induced by label inaccuracy. Extensive experiments on tasks with various inaccuracy conditions demonstrate that LIP consistently enhances the performance of existing algorithms. We hope our findings can offer valuable theoretical and practical insights to understand of model robustness under inaccurate supervision.

Figures (4)

• Figure 1: Singular values under different label inaccuracy (i.e., $p$). We also provide the corresponding classification accuracy.
• Figure 2: Subspace similarity between top-$i$ column vectors of $\mathbf{W}$ and top-$j$ of $\mathbf{W}'$. As label inaccuracy increases, the weight matrix is able to preserve the characteristics of the principal subspace within a certain range of inaccuracy, thus retaining the most important task-specific information. However, beyond this range, the principal subspace of the weights begins to change, and even eventually becomes completely uncorrelated, leading to the loss of critical information. For clearer illustrations, we present the analysis for $i,j \in [1,60]$.
• Figure 3: Framework of LIP. Once a method is trained, the LIP applies post-processing to the trained weights used for classification. The two modules of the LIP, Principle Subspace Preservation and Label Ambiguity Purification, are utilized to retain critical information in the principal subspace and to rectify noise resulting from label inaccuracy, respectively. Given the high processing speed and extreme lightness of these modules, coupled with the absence of a need for training, LIP emerges as an efficient and lightweight plug-in.
• Figure 4: Sensitivity analysis on $k$. LIP is coupled with PRODEN on CUB-200 dataset.