Do Multilingual LLMs have specialized language heads?
Muhammad Naufil
TL;DR
The paper investigates whether multilingual LLMs have language-specific attention heads and whether unwanted languages can be pruned without harming target-language performance. It analyzes the Cohere Aya-23 8B transformer across 23 languages, employing per-head gating and head masking to classify attention heads into English-specific, Hindi-specific, language-agnostic, and miscellaneous categories, with evaluation aided by GPT-3.5-Turbo as a semantic judge. The study finds distinct head types and demonstrates that selective pruning can reduce deployment complexity while preserving accuracy for chosen languages, contributing to mechanistic interpretability in multilingual NLP. These findings offer practical guidance for language-targeted deployment and motivate further cross-language analyses with broader language coverage.
Abstract
Multilingual large language models (LLMs) have gained significant popularity for their ability to process and generate text across multiple languages. However, deploying these models in production can be inefficient when only a subset of the supported languages is of interest. There has been some research conducted on identifying whether machine translation models have language-specific or language-agnostic heads, however no research has been conducted for multilingual LLMs, to the best of our knowledge, that as we know are capable of performing diverse tasks beyond just translation. This paper explores whether multilingual LLMs have specialized language attention heads for each language, and investigates the possibility of removing language-specific heads for unwanted languages without degrading performance in the targeted languages. Our findings could inform more efficient deployment strategies for multilingual LLMs, enabling reduced model complexity while maintaining high accuracy for targeted languages.