One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?

TL;DR

The paper presents a novel benchmark for persona-instruction alignment across 12 Indic languages, evaluating four retrieval-related tasks with a frozen-encoder setup across eight multilingual models. It demonstrates that instruction-tuned embeddings excel in retrieval tasks while reverse retrieval benefits from multi-vector architectures, and it highlights cross-lingual transfer challenges driven by script boundaries. The work provides reproducible baselines and analysis of cross-lingual transfer patterns, offering practical guidance for model selection in Indic multilingual retrieval and exposing calibration trade-offs in compatibility classification. Overall, it advances understanding of how well embeddings encode persona-instruction compatibility in low-resource Indian languages and lays groundwork for improving persona-grounded, multilingual AI systems.

Abstract

Aligning multilingual assistants with culturally grounded user preferences is essential for serving India's linguistically diverse population of over one billion speakers across multiple scripts. However, existing benchmarks either focus on a single language or conflate retrieval with generation, leaving open the question of whether current embedding models can encode persona-instruction compatibility without relying on response synthesis. We present a unified benchmark spanning 12 Indian languages and four evaluation tasks: monolingual and cross-lingual persona-to-instruction retrieval, reverse retrieval from instruction to persona, and binary compatibility classification. Eight multilingual embedding models are evaluated in a frozen-encoder setting with a thin logistic regression head for classification. E5-Large-Instruct achieves the highest Recall@1 of 27.4\% on monolingual retrieval and 20.7\% on cross-lingual transfer, while BGE-M3 leads reverse retrieval at 32.1\% Recall@1. For classification, LaBSE attains 75.3\% AUROC with strong calibration. These findings offer practical guidance for model selection in Indic multilingual retrieval and establish reproducible baselines for future work\footnote{Code, datasets, and models are publicly available at https://github.com/aryashah2k/PI-Indic-Align.

Figures (4)

• Figure 1: Overview of the four benchmark tasks. (A) T1: monolingual retrieval. (B) T2: cross-lingual retrieval. (C) T3: reverse retrieval. (D) T4: compatibility classification. Nodes represent the 12 evaluated Indian languages. Takeaway: The benchmark evaluates both retrieval directions and binary classification, enabling comprehensive assessment of embedding quality for persona-instruction alignment.
• Figure 2: Dataset construction: (1) persona schema adaptation, (2) English synthesis with GPT-4o-mini, (3) translation via NLLB-200, (4) human validation with two annotators per language. Takeaway: The pipeline ensures both linguistic diversity (12 languages, 10 scripts) and quality control through native-speaker validation.
• Figure 3: Model performance across retrieval tasks (T1--T3). Takeaway: Instruction-tuned models (E5, BGE-M3) consistently outperform general-purpose multilingual encoders by 10+ percentage points.
• Figure 4: Calibration reliability diagram for T4. Takeaway: High-accuracy models (E5, BGE-M3) show overconfidence, while DistilUSE achieves best calibration despite lower discriminative performance.