Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition

TL;DR

Analysis of Architectural Fingerprinting reveals divergent hierarchies: Conformers' front-loaded categorization may benefit low-latency streaming, while Transformers'deep integration may favor tasks requiring rich context and cross-utterance normalization.

Abstract

In speech language modeling, two architectures dominate the frontier: the Transformer and the Conformer. However, it remains unknown whether their comparable performance stems from convergent processing strategies or distinct architectural inductive biases. We introduce Architectural Fingerprinting, a probing framework that isolates the effect of architecture on representation, and apply it to a controlled suite of 24 pre-trained encoders (39M-3.3B parameters). Our analysis reveals divergent hierarchies: Conformers implement a "Categorize Early" strategy, resolving phoneme categories 29% earlier in depth and speaker gender by 16% depth. In contrast, Transformers "Integrate Late," deferring phoneme, accent, and duration encoding to deep layers (49-57%). These fingerprints suggest design heuristics: Conformers' front-loaded categorization may benefit low-latency streaming, while Transformers' deep integration may favor tasks requiring rich context and cross-utterance normalization.

Figures (3)

• Figure 1: t-SNE visualization of encoder representations comparing a Transformer (Whisper-large-v3-turbo, top row) and a Conformer (Canary-1B, bottom row) across network depth. Each column shows representations at increasing depths (0%, 37.5%, 75%, and 100% of total layers). The first three columns are colored by L1 background: Arabic (gray), Hindi (blue), Korean (green), Mandarin (red), Spanish (magenta), and Vietnamese (orange). The rightmost column (100% depth) uses a distinct color palette, colored by utterance identity (12 discrete colors for 12 different utterances), to reveal how content clusters in the final layer independent of L1 background. Utterance colors are assigned arbitrarily and do not correspond to specific L1 backgrounds. Points are plotted in random order within each depth column to avoid visual occlusion bias. The Transformer shows gradual emergence of accent clustering in deep layers, while the Conformer exhibits earlier and sharper accent separation, consistent with their divergent processing hierarchies.
• Figure 2: Mean peak layer position (as percentage of model depth) for five feature categories across Transformer (N=17) and Conformer (N=7) architectures. Error bars indicate standard error of the mean. The architectural divergence is clear: Conformers front-load Gender and Phoneme (both $<$ 25% depth) while deferring Duration to final layers ($\sim$70% depth), creating a distinct hierarchy. Transformers compress all high-level features into a narrow band (49--57% depth), reflecting their integrated processing strategy.
• Figure 3: Smoothed layer-wise probing trajectories across normalized depth for representative feature groups (Acoustic, Gender, Accent, Phoneme), shown separately for Transformers (left) and Conformers (right). Curves are LOWESS fits with 95% bootstrapped confidence intervals. To emphasize where features become accessible (rather than absolute probe difficulty), probing scores within each feature group are min--max normalized before smoothing. Duration is omitted from this visualization for clarity; its peak positions are reported in Figure \ref{['fig:peak_positions']} and the main text.