Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark

Daniil Gurgurov, Yusser Al Ghussin, Tanja Baeumel, Cheng-Ting Chou, Patrick Schramowski, Marius Mosbach, Josef van Genabith, Simon Ostermann

TL;DR

CLaS-Bench provides the first standardized benchmark for cross-lingual language steering, addressing a gap where prior work focused mainly on English. It evaluates multilingual and cross-lingual steering across 32 languages using 70 parallel prompts per language, with two orthogonal metrics—Language Forcing and Output Relevance—combined into a harmonic Language Steering Score. The study finds that residual DiffMean steering on hidden activations consistently outperforms prompting and other methods, while language-specific neuron and SAE-based approaches show more limited, language-dependent performance and sometimes higher variance. Layer-wise analysis reveals language information concentrates in later transformer layers, and languages from related families cluster geometrically in representation space, informing both interpretability and practical cross-lingual adaptation. Overall, CLaS-Bench enables rigorous, scalable investigation of multilingual representations and steerable control, facilitating low-cost, language-aware model adaptation and deeper mechanistic understanding of multilingual LLMs.

Abstract

Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, , i.e.,~manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of steering techniques. We introduce CLaS-Bench, a lightweight parallel-question benchmark for evaluating language-forcing behavior in LLMs across 32 languages, enabling systematic evaluation of multilingual steering methods. We evaluate a broad array of steering techniques, including residual-stream DiffMean interventions, probe-derived directions, language-specific neurons, PCA/LDA vectors, Sparse Autoencoders, and prompting baselines. Steering performance is measured along two axes: language control and semantic relevance, combined into a single harmonic-mean steering score. We find that across languages simple residual-based DiffMean method consistently outperforms all other methods. Moreover, a layer-wise analysis reveals that language-specific structure emerges predominantly in later layers and steering directions cluster based on language family. CLaS-Bench is the first standardized benchmark for multilingual steering, enabling both rigorous scientific analysis of language representations and practical evaluation of steering as a low-cost adaptation alternative.

CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark

TL;DR

CLaS-Bench provides the first standardized benchmark for cross-lingual language steering, addressing a gap where prior work focused mainly on English. It evaluates multilingual and cross-lingual steering across 32 languages using 70 parallel prompts per language, with two orthogonal metrics—Language Forcing and Output Relevance—combined into a harmonic Language Steering Score. The study finds that residual DiffMean steering on hidden activations consistently outperforms prompting and other methods, while language-specific neuron and SAE-based approaches show more limited, language-dependent performance and sometimes higher variance. Layer-wise analysis reveals language information concentrates in later transformer layers, and languages from related families cluster geometrically in representation space, informing both interpretability and practical cross-lingual adaptation. Overall, CLaS-Bench enables rigorous, scalable investigation of multilingual representations and steerable control, facilitating low-cost, language-aware model adaptation and deeper mechanistic understanding of multilingual LLMs.

Abstract

Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, , i.e.,~manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of steering techniques. We introduce CLaS-Bench, a lightweight parallel-question benchmark for evaluating language-forcing behavior in LLMs across 32 languages, enabling systematic evaluation of multilingual steering methods. We evaluate a broad array of steering techniques, including residual-stream DiffMean interventions, probe-derived directions, language-specific neurons, PCA/LDA vectors, Sparse Autoencoders, and prompting baselines. Steering performance is measured along two axes: language control and semantic relevance, combined into a single harmonic-mean steering score. We find that across languages simple residual-based DiffMean method consistently outperforms all other methods. Moreover, a layer-wise analysis reveals that language-specific structure emerges predominantly in later layers and steering directions cluster based on language family. CLaS-Bench is the first standardized benchmark for multilingual steering, enabling both rigorous scientific analysis of language representations and practical evaluation of steering as a low-cost adaptation alternative.
Paper Structure (48 sections, 8 equations, 23 figures, 15 tables)

This paper contains 48 sections, 8 equations, 23 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Benchmark Design
  3. Languages.
  4. Evaluation data.
  5. Task definition.
  6. Evaluation metrics.
  7. Experimental Setup
  8. Models.
  9. Steering methods.
  10. Steering Results
  11. Residual-based steering outperforms prompting.
  12. SAE-based steering lags behind.
  13. Supervised methods underperform unsupervised approaches.
  14. Ablation Analysis
  15. Layer and steering strength interact.
  16. ...and 33 more sections

Figures (23)

  • Figure 1: CLaS-Bench pipeline: Multilingual inputs consisting of 70 parallel questions (Q) across 32 languages (L) are evaluated per target language. Each input is passed to an LLM, which is steered with a selected method. The steered model output is evaluated along two axes: language forcing F (whether generation switches to the intended target language) and output relevance R (whether response is related to the input). These metrics are combined via a harmonic mean into a single steering score S.
  • Figure 2: Analysis of steering methods across evaluation metrics for Llama-3.1-8B-Instruct. Columns show different methods. Rows represent: forcing success rate, judge relevance, and overall steering score.
  • Figure 3: Insights into language-specific components across interpretation tools for Llama-3.1-8B-Instruct. (a) reveals average cosine similarity patterns across all language vectors. (b) demonstrates probe learning dynamics through loss and accuracy trajectories. (c) identifies the distribution of language-specific neurons across layers. (d) provides LDA classification accuracy and Fisher Ratio (the degree of separability between classes).
  • Figure 4: Distribution of LAPE identified language-specific neurons over layers in Llama-3.1-Instruct for all 32 languages.
  • Figure 5: Distribution of LAPE identified language-specific neurons over layers in Aya-Expanse-8B for all 32 languages.
  • ...and 18 more figures