A Shared Geometry of Difficulty in Multilingual Language Models

TL;DR

This work investigates how multilingual LLMs internalize problem difficulty and whether such signals transfer across languages. The authors build a multilingual AMC Easy2Hard benchmark across $21$ languages, extract residual transformer activations, and train per-layer linear probes with Ridge regression to predict continuous difficulty scores in $[0,1]$ using Spearman correlation $\\rho$ as the evaluation metric. They find a two-stage geometry: a language-agnostic difficulty direction emerging in shallow layers that transfers well across languages, and a language-specific refinement in deeper layers that boosts within-language accuracy but harms cross-lingual alignment. These results extend prior English-focused findings to a multilingual setting and suggest practical applications for language-agnostic routing and curriculum design in multilingual systems, while highlighting that early difficulty signals are robust across typologically diverse languages. The study also demonstrates that cross-lingual transfer peaks at earlier network stages, whereas monolingual performance improves at deeper layers, revealing a fundamental trade-off in how meta-cognitive properties are represented inside LLMs.

Abstract

Predicting problem-difficulty in large language models (LLMs) refers to estimating how difficult a task is according to the model itself, typically by training linear probes on its internal representations. In this work, we study the multilingual geometry of problem-difficulty in LLMs by training linear probes using the AMC subset of the Easy2Hard benchmark, translated into 21 languages. We found that difficulty-related signals emerge at two distinct stages of the model internals, corresponding to shallow (early-layers) and deep (later-layers) internal representations, that exhibit functionally different behaviors. Probes trained on deep representations achieve high accuracy when evaluated on the same language but exhibit poor cross-lingual generalization. In contrast, probes trained on shallow representations generalize substantially better across languages, despite achieving lower within-language performance. Together, these results suggest that LLMs first form a language-agnostic representation of problem difficulty, which subsequently becomes language-specific. This closely aligns with existing findings in LLM interpretability showing that models tend to operate in an abstract conceptual space before producing language-specific outputs. We demonstrate that this two-stage representational process extends beyond semantic content to high-level meta-cognitive properties such as problem-difficulty estimation.

Figures (4)

• Figure 1: Layer-wise performance of difficulty probes across languages for LLaMA-3.1-8B. Heatmap shows, for each test language (rows) and transformer layer (columns), the average Spearman correlation between probe predictions and ground-truth difficulty, where each value is averaged over probes trained on all other languages. Performance peaks in the middle layers, indicating that difficulty representations are most consistently aligned across languages at intermediate depths of the network.
• Figure 2: Cross-lingual structure of difficulty representations in LLaMA-3.1-8B.Left: Maximum Spearman $\rho$ achieved by linear difficulty probes for each training--testing language pair, evaluated at the layer that maximizes performance for that pair. Diagonal entries correspond to same-language probing, while off-diagonal entries reflect cross-lingual transfer. Right: Transformer layer indices at which peak performance is attained for each language pair.
• Figure 3: Layer-wise cross-lingual difficulty probing for additional models. Same setup as Figure \ref{['fig:layerwise-transfer']}, but for LLaMA-3.2-3B, LLaMA-3.2-1B, and Qwen3-8B. Heatmaps report, for each test language and transformer layer, the average Spearman correlation between predicted and ground-truth difficulty, averaged over probes trained on all other languages. As in Figure \ref{['fig:layerwise-transfer']}, cross-lingual performance peaks in early-to-middle layers across models.
• Figure 4: Cross-lingual structure of difficulty representations for additional models. Same analysis as Figure \ref{['fig:spearman_matrices_Llama3_1']} but for LLaMA-3.2-3B, LLaMA-3.2-1B, and Qwen3-8B. Left: maximum Spearman $\rho$ for each training–testing language pair at its optimal layer. Right: corresponding layer indices achieving peak performance. The same pattern holds across models, with cross-lingual optima concentrated at earlier layers than monolingual optima.