One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations

TL;DR

Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), it is found that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines, suggesting SAE features prioritize meaning over orthographic form.

Abstract

Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.

Figures (5)

• Figure 1: Baseline semantic discrimination across model scales. Larger models achieve greater separation between paraphrase and random similarity, with Serbian Latin and Serbian Cyrillic following nearly parallel trajectories.
• Figure 2: Cross-script similarity across model scales. Cross-script original and paraphrase similarity increase with model size while cross-script random decreases, indicating increasingly robust script-invariant representations. Notably, cross-script cross-paraphrase remains stable across scales.
• Figure 3: Mean token counts by script and sentence type.
• Figure 4: Token count difference vs. SAE feature Jaccard similarity.
• Figure 5: LaBSE sentence similarity verification. (a) Histogram of similarity scores showing clear separation between random pairs and semantically related pairs. (b) Box plots of similarity distributions by condition. Cross-script original pairs achieve near-ceiling similarity, confirming semantic equivalence across scripts.