Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

Sungmok Jung, Yeonkyoung So, Joonhak Lee, Sangho Kim, Yelim Ahn, Jaejin Lee

TL;DR

This work shows that Korean negation poses a substantial challenge to current LLMs. It first analyzes Korean negation empirically to reveal distributional properties and then introduces Thunder-KoNUBench, a corpus-aligned, sentence-level MCQA benchmark that mirrors these properties. Evaluating 47 LLMs, the study demonstrates that larger models are generally more robust to negation, but multilingual instruction tuning can hurt performance in low-resource settings; importantly, cloze-style supervised fine-tuning on Thunder-KoNUBench yields the strongest improvements in negation understanding and broader contextual reasoning. The benchmark, its construction protocol, and the accompanying insights offer a practical tool for evaluating and improving Korean negation understanding in multilingual AI systems, with broader implications for language-agnostic negation evaluation.

Abstract

Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding, especially in Korean, are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs, we analyze the effects of model size and instruction tuning, and show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

TL;DR

This work shows that Korean negation poses a substantial challenge to current LLMs. It first analyzes Korean negation empirically to reveal distributional properties and then introduces Thunder-KoNUBench, a corpus-aligned, sentence-level MCQA benchmark that mirrors these properties. Evaluating 47 LLMs, the study demonstrates that larger models are generally more robust to negation, but multilingual instruction tuning can hurt performance in low-resource settings; importantly, cloze-style supervised fine-tuning on Thunder-KoNUBench yields the strongest improvements in negation understanding and broader contextual reasoning. The benchmark, its construction protocol, and the accompanying insights offer a practical tool for evaluating and improving Korean negation understanding in multilingual AI systems, with broader implications for language-agnostic negation evaluation.

Abstract

Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding, especially in Korean, are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs, we analyze the effects of model size and instruction tuning, and show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.
Paper Structure (50 sections, 4 figures, 17 tables)

This paper contains 50 sections, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges of LLMs in Negation Handling
  4. Datasets and Benchmarks for Negation
  5. Korean Negation
  6. Definition of Negation in Korean
  7. Standard Negation.
  8. Local Negation.
  9. Statistics of Korean Negation
  10. LLMs on Korean negation
  11. LLM performance on KMMLU.
  12. LLM performance on KoBest BoolQ.
  13. Thunder-KoNUBench
  14. Task Overview
  15. Distributions of negation.
  16. ...and 35 more sections

Figures (4)

  • Figure 1: Illustration of negation-induced performance evaluation on KMMLU and KoBest BoolQ.
  • Figure 2: An instance of Thunder-KoNUBench.
  • Figure 3: Model performance across different model sizes on zero-shot setting. The horizontal axis represents model size (in billions of parameters), and the vertical axis indicates performance (acc or acc_norm).
  • Figure 4: Performance change of instruction-tuned models relative to their base counterparts. The vertical axis represents the difference between the performance of instruction-tuned models and that of the corresponding base models.