Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

Donghyeon Ko, Yeguk Jin, Kyubyung Chae, Byungwook Lee, Chansong Jo, Sookyo In, Jaehong Lee, Taesup Kim, Donghyun Kwak

TL;DR

KoSimpleQA fills a gap in factuality benchmarking by focusing on Korean cultural knowledge with 1,000 unambiguous questions collected under strict temporal and difficulty constraints. The authors evaluate a broad set of open-source LLMs that support Korean and analyze how reasoning-enabled models behave on factual QA, finding that even strong models struggle (max ~33.7% accuracy) and that cultural grounding affects model rankings relative to English benchmarks. A key contribution is the demonstration that reasoning can both reveal latent knowledge and increase abstention when uncertain, highlighting nuanced dynamics in factual QA. The dataset and findings provide a foundation for culturally grounded, factually reliable language models and point to future work in extending evaluation to longer, more complex outputs.

Abstract

We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at https://anonymous.4open.science/r/KoSimpleQA-62EB.

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

TL;DR

KoSimpleQA fills a gap in factuality benchmarking by focusing on Korean cultural knowledge with 1,000 unambiguous questions collected under strict temporal and difficulty constraints. The authors evaluate a broad set of open-source LLMs that support Korean and analyze how reasoning-enabled models behave on factual QA, finding that even strong models struggle (max ~33.7% accuracy) and that cultural grounding affects model rankings relative to English benchmarks. A key contribution is the demonstration that reasoning can both reveal latent knowledge and increase abstention when uncertain, highlighting nuanced dynamics in factual QA. The dataset and findings provide a foundation for culturally grounded, factually reliable language models and point to future work in extending evaluation to longer, more complex outputs.

Abstract

We present , a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at https://anonymous.4open.science/r/KoSimpleQA-62EB.
Paper Structure (23 sections, 3 figures, 4 tables)

This paper contains 23 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. KoSimpleQA
  3. Data Collection
  4. Category-guided annotation.
  5. Cultural grounding.
  6. Temporal constraint.
  7. Difficulty requirement.
  8. Unambiguous answers.
  9. Source verification.
  10. Quality Control
  11. Metrics
  12. Experiments
  13. Setting
  14. Baselines
  15. Sampling Parameters
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The importance of cultural knowledge for LLMs within a specific language community: an LLM with only linguistic competence responds fluently in Korean but fails, whereas one with Korean cultural knowledge correctly answers the question “Who is the person on the 50,000 won bill?”
  • Figure 2: Distribution of categories in KoSimpleQA. The legend was ordered by sample size, from largest to smallest.
  • Figure 3: The rankings of the largest models in each LLM family on SimpleQA and KoSimpleQA.