TRACE for Tracking the Emergence of Semantic Representations in Transformers

TL;DR

TRACE proposes that linguistic abstractions in transformer language models emerge at identifiable training milestones rather than gradually. It combines three signals—loss-landscape geometry via Hessian curvature, intrinsic dimensionality of representations, and linguistic-category alignment from probing and generation—using ABSynth25K, a frame-semantic synthetic corpus with controllable complexity. The authors introduce a curvature complexity measure $\mathcal{C}(H)=\frac{\mathrm{Tr}(H)}{\sqrt{r_{eff}}}$ and demonstrate coordinated phase transitions across model scales and ablations, where curvature flattens, ID stabilises, and linguistic alignment improves. This framework offers insights into interpretability, training efficiency, and compositional generalisation, suggesting practical markers for principled LM development.

Abstract

Modern transformer models exhibit phase transitions during training, distinct shifts from memorisation to abstraction, but the mechanisms underlying these transitions remain poorly understood. Prior work has often focused on endpoint representations or isolated signals like curvature or mutual information, typically in symbolic or arithmetic domains, overlooking the emergence of linguistic structure. We introduce TRACE (Tracking Representation Abstraction and Compositional Emergence), a diagnostic framework combining geometric, informational, and linguistic signals to detect phase transitions in Transformer-based LMs. TRACE leverages a frame-semantic data generation method, ABSynth, that produces annotated synthetic corpora with controllable complexity, lexical distributions, and structural entropy, while being fully annotated with linguistic categories, enabling precise analysis of abstraction emergence. Experiments reveal that (i) phase transitions align with clear intersections between curvature collapse and dimension stabilisation; (ii) these geometric shifts coincide with emerging syntactic and semantic accuracy; (iii) abstraction patterns persist across architectural variants, with components like feedforward networks affecting optimisation stability rather than fundamentally altering trajectories. This work advances our understanding of how linguistic abstractions emerge in LMs, offering insights into model interpretability, training efficiency, and compositional generalisation that could inform more principled approaches to LM development.

Figures (12)

• Figure 1: Overview of the TRACE framework, which integrates the monitoring of (i) intrinsic dimensionality of hidden states, (ii) spectral curvature complexity of the loss landscape, and (iii) linguistic alignment via probing and output accuracy. Inputs are sampled from ABSynth, our proposed synthetically generated corpus grounded on frame-based representations and controlled distributions over entropy, frequency, and complexity.
• Figure 2: Coordinated dynamics of Hessian Curvature Score (blue) and Average Intrinsic Dimension (red) across training steps for three model architectures. Each row shows a different architectural variant: standard models (top), models without feed-forward networks (middle), and models with a single attention head (bottom). The intersection points between curvature and ID trajectories mark critical phase transitions in representational learning, with timing and stability varying across architectures but preserving the fundamental pattern.
• Figure 3: Probe confidence scores over training steps for the large model. Each subplot corresponds to a different decoder layer, with curves representing average model confidence for the presence of specific linguistic tags.
• Figure 4: SRL performance per label across models and training steps
• Figure 5: The frame-semantic data generation pipeline: (1) Frame selection with semantic roles, (2) Lexical realisation with Zipfian scaling, (3) Syntactic construction following grammatical constraints, (4) Entropy calibration for controlled predictability. Each generated sentence preserves ground-truth annotations from the underlying frame structure.