HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models

TL;DR

HAE-RAE Bench introduces a Korean-knowledge-focused benchmark to evaluate the depth of Korean cultural and contextual knowledge encoded in language models, addressing shortcomings of translation-based multilingual benchmarks. It comprises six downstream tasks across four domains (vocabulary, history, general knowledge, and reading comprehension) and emphasizes native Korean knowledge over generic NLU. The authors compare Korean-focused, multilingual, and English-centric models, revealing that non-Korean models struggle more on HAE-RAE than KoBEST, and that gains from cross-lingual prompting are limited for culture-specific knowledge. The work highlights the need for Korean-centric corpora and evaluation criteria to build effective Korean conversational agents and search systems, and provides public access to the benchmark for future research.

Abstract

Large language models (LLMs) trained on massive corpora demonstrate impressive capabilities in a wide range of tasks. While there are ongoing efforts to adapt these models to languages beyond English, the attention given to their evaluation methodologies remains limited. Current multilingual benchmarks often rely on back translations or re-implementations of English tests, limiting their capacity to capture unique cultural and linguistic nuances. To bridge this gap for the Korean language, we introduce the HAE-RAE Bench, a dataset curated to challenge models lacking Korean cultural and contextual depth. The dataset encompasses six downstream tasks across four domains: vocabulary, history, general knowledge, and reading comprehension. Unlike traditional evaluation suites focused on token and sequence classification or mathematical and logical reasoning, the HAE-RAE Bench emphasizes a model's aptitude for recalling Korean-specific knowledge and cultural contexts. Comparative analysis with prior Korean benchmarks indicates that the HAE-RAE Bench presents a greater challenge to non-Korean models by disturbing abilities and knowledge learned from English being transferred.

Figures (17)

• Figure 1: Example instance from the HAE-RAE Bench. English translations are added for broader accessibility.
• Figure 2: Density distribution of answer choices by Polyglot-Ko-12.8B, GPT-4, and Gold Labels.
• Figure 3: Accuracy of Polyglot-Ko-12.8B and GPT-4 on sub-categories of General Knowledge. The striped areas within each bar represent questions that both models answered correctly.
• Figure 4: Accuracy of Polyglot-Ko-12.8B and GPT-4 on different levels of Reading Comprehension.
• Figure 5: Prompt used in our Direct Evaluation.