Extending Beacon to Hindi: Cultural Adaptation Drives Cross-Lingual Sycophancy

TL;DR

This study addresses whether Beacon's English-centric sycophancy diagnostic generalizes to Hindi. It introduces a three-condition Hindi benchmark (English original, Hindi literal, and Hindi culturally adapted prompts) and evaluates four instruction-tuned models on 50 prompts per condition, enabling separation of language-encoding and cultural-adaptation effects. Across models, Hindi prompts show a 12–16 percentage-point higher sycophancy rate than English, with a decomposition attributing most of the gap to cultural adaptation (Δ = 14.0 percentage points, 95% CI: [4.0%, 26.0%]) and minimal contribution from language encoding (Δ = 2.0 percentage points, 95% CI including zero). A translation-only baseline suggests the effect is not primarily a translation artifact, though limited by sample size and single-model evaluation. The work provides a reproducible methodology for extending alignment evaluation to non-English languages and highlights the importance of culturally grounded prompt framing in multilingual safety assessments.

Abstract

Sycophancy, the tendency of language models to prioritize agreement with user preferences over principled reasoning, has been identified as a persistent alignment failure in English-language evaluations. However, it remains unclear whether such diagnostics generalize across languages and cultural contexts. We extend the Beacon single-turn forced-choice sycophancy diagnostic to Hindi through a controlled three-condition design: English original, Hindi literal translation, and Hindi culturally adapted prompts. We evaluate four open-weight instruction-tuned models on 50 prompts per condition, enabling separation of language encoding effects from cultural adaptation effects. Across all models, sycophancy rates are consistently higher for culturally adapted Hindi prompts than for English, with absolute differences ranging from 12.0 to 16.0 percentage points. A decomposition on Qwen 2.5-Coder-7B shows that cultural adaptation (delta = 14.0%, 95% CI: [4.0%, 26.0%]) accounts for the majority of this gap, while language encoding contributes minimally (delta = 2.0%, 95% CI: [0.0%, 6.0%]). Category-level analysis reveals that advice prompts exhibit the largest cross-lingual differences (20-25 percentage points), achieving statistical significance in two of four models. These findings indicate that alignment behaviors measured in English may not transfer uniformly across languages and that culturally grounded prompt framing plays a substantial role. We release all datasets and evaluation code to support replication and extension.

Figures (4)

• Figure 1: Hindi sycophancy rates across four instruction-tuned models ($n=50$ per model). All models exhibit sycophancy rates between 16% and 22%, with Llama 3 showing the highest rate.
• Figure 2: Cross-lingual comparison of sycophancy rates. All four models exhibit consistently higher sycophancy in Hindi (dark bars) than in English (light bars), with absolute differences ranging from 12 to 16 percentage points.
• Figure 3: Cross-lingual deltas (Hindi minus English) with statistical significance indicators. Positive values indicate higher sycophancy in Hindi. Three of four models show statistically significant differences ($p < 0.05$).
• Figure 4: Category-wise sycophancy rates across models and languages. The advice category exhibits the largest cross-lingual differences, particularly for Qwen and Mistral. Darker shading indicates higher sycophancy rates.