Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

Alexey Tikhonov, Max Ryabinin

TL;DR

The paper tackles cross-lingual commonsense reasoning by proposing a lightweight, supervision-efficient baseline that trains a logistic classifier on multi-head attention weights from pretrained multilingual encoders. It introduces XWINO, a multilingual Winograd Schema corpus spanning six languages, and demonstrates that the approach achieves competitive performance in zero-shot settings, outperforming several unsupervised baselines. A core finding is that a small subset of attention heads—primarily in higher layers—drives universal commonsense reasoning across languages, suggesting language-agnostic linguistic functions emerge in multilingual models. These results highlight the potential of head-focused representations for cross-lingual reasoning and provide practical guidance for head selection in future unsupervised or low-resource settings.

Abstract

Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English. Pretrained cross-lingual models are a source of powerful language-agnostic representations, yet their inherent reasoning capabilities are still actively studied. In this work, we design a simple approach to commonsense reasoning which trains a linear classifier with weights of multi-head attention as features. To evaluate this approach, we create a multilingual Winograd Schema corpus by processing several datasets from prior work within a standardized pipeline and measure cross-lingual generalization ability in terms of out-of-sample performance. The method performs competitively with recent supervised and unsupervised approaches for commonsense reasoning, even when applied to other languages in a zero-shot manner. Also, we demonstrate that most of the performance is given by the same small subset of attention heads for all studied languages, which provides evidence of universal reasoning capabilities in multilingual encoders.

It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

TL;DR

The paper tackles cross-lingual commonsense reasoning by proposing a lightweight, supervision-efficient baseline that trains a logistic classifier on multi-head attention weights from pretrained multilingual encoders. It introduces XWINO, a multilingual Winograd Schema corpus spanning six languages, and demonstrates that the approach achieves competitive performance in zero-shot settings, outperforming several unsupervised baselines. A core finding is that a small subset of attention heads—primarily in higher layers—drives universal commonsense reasoning across languages, suggesting language-agnostic linguistic functions emerge in multilingual models. These results highlight the potential of head-focused representations for cross-lingual reasoning and provide practical guidance for head selection in future unsupervised or low-resource settings.

Abstract

Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English. Pretrained cross-lingual models are a source of powerful language-agnostic representations, yet their inherent reasoning capabilities are still actively studied. In this work, we design a simple approach to commonsense reasoning which trains a linear classifier with weights of multi-head attention as features. To evaluate this approach, we create a multilingual Winograd Schema corpus by processing several datasets from prior work within a standardized pipeline and measure cross-lingual generalization ability in terms of out-of-sample performance. The method performs competitively with recent supervised and unsupervised approaches for commonsense reasoning, even when applied to other languages in a zero-shot manner. Also, we demonstrate that most of the performance is given by the same small subset of attention heads for all studied languages, which provides evidence of universal reasoning capabilities in multilingual encoders.

Paper Structure

This paper contains 32 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Winograd Schema challenges
  4. Language models applied to commonsense reasoning
  5. Cross-lingual encoder models
  6. Functions of Transformer heads
  7. Common sense from attention
  8. Exact task specification
  9. Transformers for sentence representations
  10. Our approach
  11. Feature combination:
  12. Pooling over tokens:
  13. Attention direction:
  14. Dataset
  15. Languages
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Example of a Winograd Schema problem. The resolved pronoun is underlined, two options are highlighted with an italic font.
  • Figure 2: Averaged attention from the pronoun when using top-5 common heads.
  • Figure 3: Effect of the number of XLM-R attention heads used when training on English data. Shaded areas show standard deviation across runs.
  • Figure 4: Effect of the number of used XLM-R attention heads on commonsense reasoning performance.