Human-in-the-loop: Towards Label Embeddings for Measuring Classification Difficulty

TL;DR

This work tackles the problem of annotation-driven uncertainty when ground-truth labels are unreliable or unavailable. It introduces label embeddings by modeling per-instance annotations with a Dirichlet-Multinomial/Beta-Binomial generative framework and estimating latent embeddings $\boldsymbol{Z}_i\in\mathbb{R}^K$ via a stochastic EM algorithm that leverages MCMC, enabling a distributional representation of labels. The method is demonstrated on ChaosSNLI, So2Sat LCZ42, and CIFAR-10H, showing that the resulting embeddings capture class correlations and annotation-driven uncertainty, effectively serving as generalized confusion matrices. The findings suggest pathways to train models on embedding-based labels for better uncertainty calibration and inform annotation design, highlighting practical impact for multi-annotator settings across NLP and computer vision tasks.

Abstract

Uncertainty in machine learning models is a timely and vast field of research. In supervised learning, uncertainty can already occur in the first stage of the training process, the annotation phase. This scenario is particularly evident when some instances cannot be definitively classified. In other words, there is inevitable ambiguity in the annotation step and hence, not necessarily a "ground truth" associated with each instance. The main idea of this work is to drop the assumption of a ground truth label and instead embed the annotations into a multidimensional space. This embedding is derived from the empirical distribution of annotations in a Bayesian setup, modeled via a Dirichlet-Multinomial framework. We estimate the model parameters and posteriors using a stochastic Expectation Maximization algorithm with Markov Chain Monte Carlo steps. The methods developed in this paper readily extend to various situations where multiple annotators independently label instances. To showcase the generality of the proposed approach, we apply our approach to three benchmark datasets for image classification and Natural Language Inference. Besides the embeddings, we can investigate the resulting correlation matrices, which reflect the semantic similarities of the original classes very well for all three exemplary datasets.

Figures (13)

• Figure 1: The figure shows the categories of the LCZ classification scheme (Stewart:2012), along with exemplary images for each class from three different sources (top row: Sentinel 1, middle row: Sentinel 2, bottom row: Google Earth), images taken from Zhu:2020.
• Figure 2: The figure shows exemplary images from Cifar-10H, where a high disagreement rate between the annotators could be observed hinting at the ambiguity of the images.
• Figure 3: The figure shows the mean (left) and log-variance (right) of the Beta-Binomial distribution for different values of $\boldsymbol{Z}$, expressed through color.
• Figure 4: Estimating the embeddings.
• Figure 5: The plots show the estimated embedded ground truth vectors for exemplary sentence pairs from the dataset ChaosSNLI. The actual estimated vector is shown as orange line, the green lines represent the MCMC samples and the actual annotations are shown as grey bars.