KoBALT: Korean Benchmark For Advanced Linguistic Tasks

TL;DR

KoBALT addresses the lack of linguistically rigorous benchmarks for Korean by introducing 700 linguist-crafted MCQs spanning 24 phenomena across syntax, semantics, pragmatics, phonetics/phonology, and morphology. The benchmark minimizes data contamination through original items with limited overlap with training data and uses a 10-choice format to emphasize deep knowledge and reasoning. Evaluation across 20 LLMs reveals domain-specific performance gaps (best overall $61\%$; semantics $66\%$ vs phonology $31\%$ and morphology $36\%$), and larger models tend to perform better, though not uniformly across domains. A human-preference study with 95 annotators shows a strong alignment between KoBALT scores and human judgments ($r=0.638$ for the top model), validating KoBALT as an effective discriminator of genuine Korean linguistic competence and illustrating its potential as a framework for linguistically-grounded benchmarks in other languages.

Abstract

We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61\% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66\%) to considerable weaknesses in phonology (31\%) and morphology (36\%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models.

Figures (6)

• Figure 1: Distribution of samples across linguistic domains in KoBALT
• Figure 2: Example prompt used for model response generation. Models were instructed to select from ten possible choices after analyzing the linguistic task. The strict output format facilitated automated evaluation of model responses.
• Figure 3: Accuracy per each linguistic phenomena
• Figure 4: Human preference evaluation process utilizing the Online Live Chatbot Area concept
• Figure 5: Screenshot of the full instruction given to participants in human preference collection stage.