SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

TL;DR

The paper tackles the challenge that multilingual neural metrics for summarization often underperform and vary across languages due to internal pivoting around English. It proposes inference-time activation steering, using vector directions or linear mappings, to align representations with English and improve correlations with human judgments across eight languages. Empirical results show consistent, sometimes substantial, gains for both encoder-based and LLM-based metrics, with the magnitude of improvement depending on the language, metric, and steering method. The findings highlight the practical value of test-time interventions for multilingual evaluation and point to future work on hybrid and language-aware steering strategies to further boost cross-lingual generalization.

Abstract

An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization. A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress. Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance. Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments. We experiment with encoder- and decoder-based metrics and find that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages.

Figures (24)

• Figure 1: Illustration of our method, where activation steering induces stronger correlations between LLM-as-a-judge and human evaluation.
• Figure 2: Illustration of our SteerEval method. We first use parallel data to learn cross-lingual vectors or mappings from X $\rightarrow$ English. We then apply the steering to neural metrics. Finally, we show that such steering results in better correlation with human judgments.
• Figure 3: Relative improvement in Pearson correlation after steering for COMET.
• Figure 4: Relative improvement in Pearson correlation after steering for coherence for LLM-as-a-judge metrics. Values above 600 (shown with a black dot on top of the bar) are clipped to improve readability.
• Figure 5: Relative improvement in Pearson correlation after steering for LLM-as-a-judge metrics for completeness. Values above 600 (shown with a black dot on top of the bar) are clipped to improve readability.