Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities

Hongseok Oh, Wonseok Hwang, Kyoung-Woon On

TL;DR

The Korean Canonical Legal Benchmark (KCL) addresses knowledge-independent evaluation of legal reasoning by pairing Korean Bar Exam questions with question-aligned precedents. It comprises two components: KCL-MCQA, a 283-question MCQ set with 1,103 precedents, and KCL-Essay, a 169-question open-ended set with 550 precedents and 2,739 rubrics, evaluated via LLM-as-a-Judge. Across 30+ models, providing precedents yields substantial gains on MCQA and reveals that reasoning-focused models can approach or exceed human baselines, though KCL-Essay remains significantly harder and demands deeper inference. The benchmark demonstrates the feasibility and value of disentangling knowledge from reasoning in a low-resource language context and outlines directions for future work, including retrieval-augmented evaluation and IRAC-aligned rubrics, with all resources openly released.

Abstract

We introduce the Korean Canonical Legal Benchmark (KCL), a benchmark designed to assess language models' legal reasoning capabilities independently of domain-specific knowledge. KCL provides question-level supporting precedents, enabling a more faithful disentanglement of reasoning ability from parameterized knowledge. KCL consists of two components: (1) KCL-MCQA, multiple-choice problems of 283 questions with 1,103 aligned precedents, and (2) KCL-Essay, open-ended generation problems of 169 questions with 550 aligned precedents and 2,739 instance-level rubrics for automated evaluation. Our systematic evaluation of 30+ models shows large remaining gaps, particularly in KCL-Essay, and that reasoning-specialized models consistently outperform their general-purpose counterparts. We release all resources, including the benchmark dataset and evaluation code, at https://github.com/lbox-kr/kcl.

Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities

TL;DR

The Korean Canonical Legal Benchmark (KCL) addresses knowledge-independent evaluation of legal reasoning by pairing Korean Bar Exam questions with question-aligned precedents. It comprises two components: KCL-MCQA, a 283-question MCQ set with 1,103 precedents, and KCL-Essay, a 169-question open-ended set with 550 precedents and 2,739 rubrics, evaluated via LLM-as-a-Judge. Across 30+ models, providing precedents yields substantial gains on MCQA and reveals that reasoning-focused models can approach or exceed human baselines, though KCL-Essay remains significantly harder and demands deeper inference. The benchmark demonstrates the feasibility and value of disentangling knowledge from reasoning in a low-resource language context and outlines directions for future work, including retrieval-augmented evaluation and IRAC-aligned rubrics, with all resources openly released.

Abstract

We introduce the Korean Canonical Legal Benchmark (KCL), a benchmark designed to assess language models' legal reasoning capabilities independently of domain-specific knowledge. KCL provides question-level supporting precedents, enabling a more faithful disentanglement of reasoning ability from parameterized knowledge. KCL consists of two components: (1) KCL-MCQA, multiple-choice problems of 283 questions with 1,103 aligned precedents, and (2) KCL-Essay, open-ended generation problems of 169 questions with 550 aligned precedents and 2,739 instance-level rubrics for automated evaluation. Our systematic evaluation of 30+ models shows large remaining gaps, particularly in KCL-Essay, and that reasoning-specialized models consistently outperform their general-purpose counterparts. We release all resources, including the benchmark dataset and evaluation code, at https://github.com/lbox-kr/kcl.
Paper Structure (27 sections, 3 figures, 6 tables)

This paper contains 27 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. KCL Dataset Construction
  3. Motivation
  4. Design
  5. KCL-MCQA
  6. KCL-Essay
  7. Experimental Setup
  8. Inference
  9. Evaluation
  10. Analysis
  11. The Impact of Legal Knowledge on Problem Solving Performance
  12. On the Importance of Reasoning for KCL-Essay
  13. On the Performance of Open-Weight Models in KCL
  14. Effects of Language Specialization
  15. Conclusion
  16. ...and 12 more sections

Figures (3)

  • Figure 1: An overview of the KCL benchmark and representative samples translated into English.
  • Figure 2: Pearson correlation heatmaps between LLMs and human expert evaluators. Whole Score represents the correlation over all question-prediction pairs derived from 641 triplets, while Low Score corresponds to the correlation for easy questions (bottom 80%), and High Score corresponds to the correlation for difficult questions (top 20%).
  • Figure 3: Histogram of score distribution across all 169 questions in KCL-Essay. The cutoff for the top 20% is observed at a score of 20.