Class-wise Autoencoders Measure Classification Difficulty And Detect Label Mistakes

TL;DR

The paper addresses the challenge of assessing classification dataset difficulty and identifying label issues without heavy model training by introducing Reconstruction Error Ratios (RERs). It trains one shallow autoencoder per class on foundation-model features and uses per-class reconstruction errors to form ratios that quantify sample- and dataset-level difficulty, while decomposing this difficulty into finite-sample and Bayes/decision-boundary contributions. Across 19 visual datasets, RERs correlate strongly with state-of-the-art error rates and enable competitive mislabel detection, even under various noise types, with efficient, scalable computation. The framework is shown to be domain-agnostic and applicable to data pruning, data collection, reannotation, and model selection, with an accompanying open-source implementation.

Abstract

We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes. This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels. We define reconstruction error ratios (RERs) that probe classification difficulty and allow its decomposition into (1) finite sample size and (2) Bayes error and decision-boundary complexity. Through systematic study across 19 popular visual datasets, we find that our RER-based dataset difficulty probe strongly correlates with error rate for state-of-the-art (SOTA) classification models. By interpreting sample-level classification difficulty as a label mistakenness score, we further find that RERs achieve SOTA performance on mislabel detection tasks on hard datasets under symmetric and asymmetric label noise. Our code is publicly available at https://github.com/voxel51/reconstruction-error-ratios.

Figures (18)

• Figure 1: Reconstruction error distributions for in-class and out-of-class samples shown for the easiest, median, and hardest classes in the CIFAR-10 dataset, as measured by the average ratio of in-class and out-of-class reconstruction errors. In all cases, both in-class and out-of-class reconstruction errors are well-approximated with normal distributions. $R^2$ is the coefficient of determination, which is computed by evaluating the Gaussian fit curve at the center of each bin for $100$-bin histograms. In-class refers to reconstruction error with the ground truth class's reconstructor; out-of-class refers to all other reconstruction errors.
• Figure 2: Visualization of $\chi$ for the easiest (left) and hardest (right) samples in CIFAR-10, using CLIP ViT-B/$32$ features used to train class reconstructors. Images generated using the Fiftyone library moore2020fiftyone.
• Figure 3: Scatterplot of SOTA classification error rate (plotted on a logarithmic scale) for $19$ popular computer vision datasets versus estimated classification difficulty $\overline{\chi}$ computed using the reconstruction error ratio method. Autoencoders are trained on CLIP ViT-L/$14$ features and default parameters detailed in Table \ref{['tab:autoencoder_hyperparameters']}. Points are colored by the number of classes, scaled logarithmically, and are sized proportionately to the number of samples in the dataset. Log-error-rate and $\overline{\chi}$ are found to have a Pearson correlation coefficient of $\rho = 0.639$, and this increases to $\rho = 0.780$ when Oxford $102$ Flowers is excluded.
• Figure 4: Dependence of dataset difficulty measure $\overline{\chi}$ (using CLIP ViT-B/$32$ features) on the number of samples per class used to train each reconstructor. We observe $\overline{\chi}_n$ to be well-behaved when $n \geq 20$, and for datasets where $\overline{\chi}_n$ does not oscillate around $1$, scaling is well approximated by rational functions of the form (\ref{['eq:chi_size_scaling']}). The infinite size limit extrapolated from this functional form is indicated by the large semi-transparent marker connected to the finite-size results by a dashed line.
• Figure 5: Relationship between $\overline{\chi}$ using CLIP ViT-B/$32$ features and symmetric, asymmetric, and confidence-based label noise for five exemplary datasets. Each point in the plot is generated by averaging over three random noise initializations.

Theorems & Definitions (1)

• proof