When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity

TL;DR

This work addresses the challenge of annotator disagreement and ambiguity in NLP by applying Mapper, a topological data analysis tool, to the embedding space of RoBERTa-Large on the MD-Offense dataset. It demonstrates that fine-tuning reorganizes embeddings into modular, non-convex regions that align with model predictions, while most connected components achieve high prediction purity even when human labels disagree. The study introduces topology-based metrics—Component Purity, Edge Agreement, and Majority Match—to quantify structure and demonstrates robustness across lenses and models, highlighting region-level decision behavior over pointwise certainty. These insights offer a complementary diagnostic to scalar metrics, with potential to guide proactive modeling strategies and annotation practices in subjective NLP tasks.

Abstract

Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over $98\%$ of connected components exhibit $\geq 90\%$ prediction purity, yet alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks.

Figures (12)

• Figure 1: Illustration of geometric (UMAP) and topological (Mapper) views of model representations. Figure(\ref{['fig:GeometryVsTopologyA']}) UMAP shows apparent separation between offensive and non-offensive instances. (\ref{['fig:GeometryVsTopologyB']}) Mapper reveals a more fragmented and connected structure, with offensive and non-offensive regions interwoven, highlighting spatial organization that UMAP flattens.
• Figure 2: Illustration of the Mapper algorithm. A high-dimensional data is first projected using a lens function (filter) to a low-dimensional space. The projection is covered by overlapping hypercubes (intervals here), and each preimage is clustered in the original space. Nodes in the resulting graph represent clusters, and edges indicate shared points due to overlap. Adapted from Figure 1 of liao_tmap_2019, available under CC BY 4.0.
• Figure 3: Mapper graphs of RoBERTa-Large [CLS] embeddings on the A++ test subset, colored by ground-truth labels (non-offensive, offensive, mixed). Both graphs use a PCA-1D lens (40 intervals, 30% overlap) and HDBSCAN clustering. Nodes are considered mixed when no single ground-truth label constitutes a majority among their points. Fine-tuning produces a more structured topology: label purity increases, fragmented components are reduced, and class-aligned subregions emerge, suggesting a modular decision space rather than a single global boundary. Note that visual proximity between nodes does not imply embedding similarity, only edges reflect topological closeness through shared overlap.
• Figure 4: Mapper graphs of [CLS] embeddings on the A0 test subset (RoBERTa-Large fine-tuned on the training set, PCA-1D lens, $r=40$, $\epsilon = 0.3$, HDBSCAN). Left: nodes colored by ground-truth labels, right: same graph colored by model prediction labels, (non-offensive, offensive, mixed). Nodes are considered mixed when no single ground-truth label constitutes a majority among their points. Although ground-truth labels appear fragmented (left), the model's predicted labels (right) exhibit smoother, regionally consistent patterns reflecting the model's binary classification training objective. Each topological component tends to predict a consistent class, with fewer intra-cluster flips. This suggests that the model performs prediction at the level of coarse topological regions, even in ambiguous data.
• Figure 5: CP across agreement levels (Base vs Fine-Tuned). Distribution of component-level purity scores using eq. \ref{['eq:component-purity']} for each agreement test subset (A0, A+, A++), aggregated across all six 1D lenses. Each point represents the purity of a connected component in a Mapper graph. Fine-tuned representations consistently increase class homogeneity within components, especially in A+ and A++, where label agreement is stronger. Full per-lens distributions are provided in Appendix \ref{['appendix:per-lens-violin-plots']}.