Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages

TL;DR

This work probes cross-lingual stability of LLM judges under controlled generation by comparing Estonian, Finnish, and Hungarian dialogues generated with identical parameters. It demonstrates that surface-level automatic metrics remain stable across languages, while discourse-level judgments such as coherence and instruction following exhibit ranking instability, revealing transfer failures rather than true model differences. The authors propose a diagnostic workflow to detect such instability before deployment and release the controlled-generation protocol, synthetic dialogues, and evaluation framework to enable replication across language families. The findings highlight the need for language-specific calibration for discourse-level evaluation in morphologically rich languages and caution against zero-shot judge transfer in practical deployments. The work provides a practical validity gate for cross-linguistic evaluation pipelines in underrepresented languages and offers tools to mitigate measurement unreliability.

Abstract

Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability. We investigate evaluation reliability by holding generation conditions constant while varying target language. Using synthetic customer-support dialogues generated with identical parameters across Estonian, Finnish, and Hungarian, we test whether automatic metrics and LLM-as-a-judge scoring produce stable model rankings across these morphologically rich, related Finno-Ugric languages. With a small set of Estonian native speaker annotations as a reference point, we find systematic ranking instabilities: surface-level metrics (lexical diversity, surface and semantic similarity) maintain cross-language stability, but pragmatic judgments (coherence, instruction-following) exhibit rank inversions and near-zero correlations. Because generation is controlled, these inconsistencies reflect how judge scoring behaves differently across languages rather than true model differences. This controlled design provides a diagnostic probe: evaluation methods that fail to maintain stability under identical generation conditions signal transfer failure before deployment. Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages, motivating language-specific calibration against targeted human baselines. We release our controlled generation protocol, synthetic data, and evaluation framework to enable replication across language families at https://github.com/isaac-chung/cross-lingual-stability-judges.

Figures (4)

• Figure 1: Example opening messages in each language from the generated dialogues. In English, it reads 'Good day! You have spoken to Klaus Customer Support, Martin here. How can I help you today?'.
• Figure 2: Cross-language ranking stability measured by Kendall's $\tau$. Error bars show 95% bootstrap confidence intervals. Numbers below bars indicate rank inversions (out of 15 possible pairwise inversions among 6 models); asterisks denote statistical significance via permutation test (* $p < 0.05$). Surface-level metrics (Grammar, Readability, Fluency) maintain high stability ($\tau \ge 0.62$) with minimal inversions. Pragmatic dimensions show systematic breakdown: Coherence exhibits near-zero or negative correlations, and LRA shows significant rank scrambling across all Finno-Ugric pairs (9*, 6*, 7* inversions). English pairs included for context, though ceiling effects limit their informativeness for Coherence.
• Figure 3: Label recovery accuracy (LRA) across categories by model for sampled dialogues in Estonian, Finnish, Hungarian, and English. Performance varies substantially by category complexity: simple binary parameters (Agent Experience, Agent Type) show consistent accuracy across all languages, while complex semantic categories (Industry: 40+ options, Problem: 20+ types) exhibit poor and inconsistent performance in all languages including English. This pattern suggests that complex parameter recovery may exceed current model capabilities regardless of target language, limiting LRA's utility as a cross-linguistic diagnostic. Unlike surface metrics and coherence assessment, where clear stability differences emerge, LRA instability appears task-dependent rather than language-dependent.
• Figure 4: Comparison between different LLM judges over Finnish dialogues across LLM-as-a-judge metrics.