IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesian

TL;DR

IndoPref addresses the shortage of native Indonesian preference data for aligning large language models with Indonesian linguistic and cultural norms. It introduces 522 human-authored prompts and 4,099 native Indonesian pairwise preferences across 10 domains, with high inter-annotator reliability, derived from comparisons among five instruction-tuned LLMs. An evaluation across 12 models shows Gemini 2.5 Pro achieving the highest average alignment with human preferences, while smaller open-weight models like mR3 4B also excel in certain tasks, and translation and safety tasks remain challenging. IndoPref provides a culturally faithful benchmark to advance Indonesian LLM alignment and supports more equitable, multilingual NLP research.

Abstract

Over 200 million people speak Indonesian, yet the language remains significantly underrepresented in preference-based research for large language models (LLMs). Most existing multilingual datasets are derived from English translations, often resulting in content that lacks cultural and linguistic authenticity. To address this gap, we introduce IndoPref, the first fully human-authored and multi-domain Indonesian preference dataset designed to evaluate the naturalness and quality of LLM-generated text. The dataset contains 522 prompts and yields 4,099 human-annotated pairwise preferences from comparisons across five instruction-tuned LLMs. All annotations are natively written in Indonesian with strong inter-annotator agreement, measured by Krippendorff's alpha. Our benchmark spans 10 diverse categories, enabling practitioners to identify LLMs' fine-grained strengths and weaknesses.

Figures (1)

• Figure 1: Model performance vs. model size on $\textcolor{black}{IndoPref}$. The plot illustrates scaling trends across various model architectures, showing that larger models generally align better with human preferences.