Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

Ivaxi Sheth, Zeno Jonke, Amin Mantrach, Saab Mansour

Abstract

As large language models are increasingly deployed across diverse real-world applications, extending automated evaluation beyond English has become a critical challenge. Existing evaluation approaches are predominantly English-focused, and adapting them to other languages is hindered by the scarcity and cost of human-annotated judgments in most languages. We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS). UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation that supports cross-lingual transfer with minimal supervision. Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements over strong baselines without requiring target-language annotations.

Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

Abstract

As large language models are increasingly deployed across diverse real-world applications, extending automated evaluation beyond English has become a critical challenge. Existing evaluation approaches are predominantly English-focused, and adapting them to other languages is hindered by the scarcity and cost of human-annotated judgments in most languages. We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS). UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation that supports cross-lingual transfer with minimal supervision. Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements over strong baselines without requiring target-language annotations.
Paper Structure (39 sections, 11 equations, 4 figures, 8 tables)

This paper contains 39 sections, 11 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM Judges
  4. Multilingual LLM-Based Judges
  5. Methodology
  6. Problem Formulation.
  7. Framework Overview.
  8. Criteria Set Generation
  9. Question Generation
  10. Criteria Set Evaluation
  11. Cross-Lingual Transfer
  12. Concept-Level Aggregation.
  13. Transfer.
  14. Experimental Setup
  15. Models.
  16. ...and 24 more sections

Figures (4)

  • Figure 1: UCS framework. The LLM evaluates an input using criteria questions derived from higher-level concepts. Criterion scores are aggregated into a latent representation, which a transfer module trained on English-labeled data maps to the final judgment and applies across languages.
  • Figure 2: Sample efficiency of training the transfer module on English data. Performance is shown as a function of the percentage of English training data used. Results are reported for MEMERAG (a) and mFACE (b).
  • Figure 3: Relationship between English–target criteria importance alignment and performance change when training with only the top-10 English-selected criteria.
  • Figure 4: Relationship between English–target criteria importance alignment.