Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models

Dongjun Kim, Chanhee Park, Chanjun Park, Heuiseok Lim

TL;DR

The paper addresses the lack of Korean-focused instruction-following benchmarks by introducing KITE, a benchmark with two variants: KITE General (427 translated English instructions) and KITE Korean (100 Korean-specific instructions). It outlines a two-stage development pipeline for the General version (translation via GPT-4o followed by expert filtering) and a dedicated process for the Korean version, including linguistic and cultural analysis and four specialized instruction types. The evaluation combines automated, rule-based scoring with human judgments and analyzes correlations with the Open Ko-LLM Leaderboard, revealing strong human-automated agreement and exposing performance gaps between general-purpose and Korean-specific models. Public release of the KITE data and code aims to advance culturally inclusive NLP and motivate similar benchmarks for other underrepresented languages.

Abstract

The instruction-following capabilities of large language models (LLMs) are pivotal for numerous applications, from conversational agents to complex reasoning systems. However, current evaluations predominantly focus on English models, neglecting the linguistic and cultural nuances of other languages. Specifically, Korean, with its distinct syntax, rich morphological features, honorific system, and dual numbering systems, lacks a dedicated benchmark for assessing open-ended instruction-following capabilities. To address this gap, we introduce the Korean Instruction-following Task Evaluation (KITE), a comprehensive benchmark designed to evaluate both general and Korean-specific instructions. Unlike existing Korean benchmarks that focus mainly on factual knowledge or multiple-choice testing, KITE directly targets diverse, open-ended instruction-following tasks. Our evaluation pipeline combines automated metrics with human assessments, revealing performance disparities across models and providing deeper insights into their strengths and weaknesses. By publicly releasing the KITE dataset and code, we aim to foster further research on culturally and linguistically inclusive LLM development and inspire similar endeavors for other underrepresented languages.

KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models

TL;DR

The paper addresses the lack of Korean-focused instruction-following benchmarks by introducing KITE, a benchmark with two variants: KITE General (427 translated English instructions) and KITE Korean (100 Korean-specific instructions). It outlines a two-stage development pipeline for the General version (translation via GPT-4o followed by expert filtering) and a dedicated process for the Korean version, including linguistic and cultural analysis and four specialized instruction types. The evaluation combines automated, rule-based scoring with human judgments and analyzes correlations with the Open Ko-LLM Leaderboard, revealing strong human-automated agreement and exposing performance gaps between general-purpose and Korean-specific models. Public release of the KITE data and code aims to advance culturally inclusive NLP and motivate similar benchmarks for other underrepresented languages.

Abstract

The instruction-following capabilities of large language models (LLMs) are pivotal for numerous applications, from conversational agents to complex reasoning systems. However, current evaluations predominantly focus on English models, neglecting the linguistic and cultural nuances of other languages. Specifically, Korean, with its distinct syntax, rich morphological features, honorific system, and dual numbering systems, lacks a dedicated benchmark for assessing open-ended instruction-following capabilities. To address this gap, we introduce the Korean Instruction-following Task Evaluation (KITE), a comprehensive benchmark designed to evaluate both general and Korean-specific instructions. Unlike existing Korean benchmarks that focus mainly on factual knowledge or multiple-choice testing, KITE directly targets diverse, open-ended instruction-following tasks. Our evaluation pipeline combines automated metrics with human assessments, revealing performance disparities across models and providing deeper insights into their strengths and weaknesses. By publicly releasing the KITE dataset and code, we aim to foster further research on culturally and linguistically inclusive LLM development and inspire similar endeavors for other underrepresented languages.
Paper Structure (31 sections, 1 equation, 3 figures, 5 tables)

This paper contains 31 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. KITE Benchmark
  4. Overview of KITE Benchmark
  5. Development of KITE General Benchmark
  6. Translation Process
  7. Filtering Process
  8. Development of KITE Korean Benchmark
  9. Linguistic and Cultural Analysis and Category Development
  10. Korean Specialized Instructions
  11. Experiments
  12. Experimental Design
  13. Model Details
  14. Evaluation Details
  15. Main Results
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Comparison of Human Evaluation and Automated KITE Scores (automated evaluation scores by the KITE Benchmarks). The left bar graph depicts the scores for KITE General, while the right bar graph illustrates the scores for KITE Korean. Each pair of bars represents the average scores assigned by human evaluators and the automated KITE Benchmarks for each model, demonstrating a strong agreement between the two evaluation methods.
  • Figure 2: Correlation Matrix Between KITE and Other Common Korean Benchmarks
  • Figure 3: Comparison of Model Performance on KITE and Human Evaluations across Different Shot Settings.