Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

TL;DR

This work investigates how multilingual LLMs utilize multi-head self-attention (MHA) by introducing Language Attention Head Importance Scores (LAHIS), a lightweight method that uses a trainable soft mask to quantify per-head language contributions with a single forward/backward pass. Experiments across three open multilingual models reveal distinct language-specific and language-general attention heads, enabling cross-lingual attention transfer and targeted mitigation of off-target language generation. The authors also propose a minimal, trainable head-mask adaptation (14–20 parameters) that improves XQuAD performance by about 5 points on average. Overall, the study enhances interpretability of multilingual processing in LLMs and offers practical, low-parameter interventions to strengthen multilingual capabilities.

Abstract

Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.

Figures (6)

• Figure 1: Illustration of attention head importance matrices obtained by LAHIS. Darker cells indicate higher importance scores. Most cells are light, showing that only a small subset of heads are highly important.
• Figure 2: PPL impact from deactivating language-specific heads. Cell $(i,j)$ shows the PPL increase on language $j$ when disabling language $i$’s heads. The dark diagonals indicate the the specificity of heads identified by LAHIS.
• Figure 3: An illustration of shifting LLM’s multilingual attention via head intervention. The vanilla model chooses "teacher" from the Hindi context, but switches to "scientist" from the Chinese context after weakening Hindi-heads or enhancing Chinese-heads.
• Figure 4: Layer-wise vocabulary projections of Aya-23-8B illustrating the multilingual attention shift.
• Figure 5: Influence of language-specific heads on multilingual prediction shifting. Given prompts with conflicting facts in two languages, enhancing heads for language A or disabling heads for B increases the model’s reliance on language A’s context, indicating that language-specific heads can steer cross-lingual attention and influence predictions.