CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

TL;DR

CLIcK presents a Korean-centric benchmark for cultural and linguistic intelligence, filling gaps left by translations and narrowTask benchmarks by compiling 1,995 QA pairs across 11 subcategories in two domains (Korean Culture and Korean Language) sourced from official exams and KIIP textbooks with fine-grained knowledge annotations. The dataset enables granular evaluation of 13 language model configurations, revealing that open-source models generally score 10–50% while proprietary models (e.g., GPT-3.5-turbo, Claude-2) perform better but still struggle in several categories, and that neither simply scaling nor additional Korean corpora reliably improve performance. Through ANOVA analyses, the authors show model scale and corpus size have limited impact on cultural-linguistic understanding in Korean, underscoring that non-English cultural knowledge remains a synchronization gap for LLMs. CLIcK’s public release and category-specific design provide a valuable resource for advancing culturally aware Korean language modeling and motivate future work on methods beyond data and size growth to bridge cross-cultural linguistic gaps.

Abstract

Despite the rapid development of large language models (LLMs) for the Korean language, there remains an obvious lack of benchmark datasets that test the requisite Korean cultural and linguistic knowledge. Because many existing Korean benchmark datasets are derived from the English counterparts through translation, they often overlook the different cultural contexts. For the few benchmark datasets that are sourced from Korean data capturing cultural knowledge, only narrow tasks such as bias and hate speech detection are offered. To address this gap, we introduce a benchmark of Cultural and Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs. CLIcK sources its data from official Korean exams and textbooks, partitioning the questions into eleven categories under the two main categories of language and culture. For each instance in CLIcK, we provide fine-grained annotation of which cultural and linguistic knowledge is required to answer the question correctly. Using CLIcK, we test 13 language models to assess their performance. Our evaluation uncovers insights into their performances across the categories, as well as the diverse factors affecting their comprehension. CLIcK offers the first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in Korean culture and language.

Figures (3)

• Figure 1: Overview of the CLIcK dataset curation and categorization process. Data is sourced from official exams and textbooks and validated by authors. The dataset is categorized into Korean Culture and Korean Language, further divided into 11 sub-categories.
• Figure 2: Portions of challenging samples encountered by models of varying sizes. The sky-blue bar represents the shared portion of KULLM, KoAlpaca, and LLaMa2-chat, while the gray bar corresponds to GPT-3.5 and Claude-2.
• Figure 3: Box-and-whisker plot of uncertainty score of challenging samples across the models. 'x' marks the mean, and the horizontal bar represents the median. For Polyglot and LLaMa, consistency rises with increasing model scale. Polyglot 1.3B has a median near 1, while LLaMa's median is closer to 0.