Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond

TL;DR

This work conducts a comprehensive, bilingual evaluation of 12 LLMs across 17 Chinese and English legal reasoning tasks to assess the value of test-time scaling for legal cognition. It introduces Legal-R1, an open-source 14B legal-specialized model trained via a progressive supervised fine-tuning pipeline on a bilingual, rejection-sampled CoT dataset distilled from DeepSeek-R1, and compares it against state-of-the-art test-time scaled models. The results show that DeepSeek-R1 remains especially strong in Chinese tasks, while o1-preview performs well on English tasks; Legal-R1 provides competitive or superior performance on several key tasks and offers a robust open-source baseline. Error analysis reveals recurring issues like outdated knowledge, misinterpretation of citations, and factual hallucinations, suggesting directions such as multilingual legal datasets and retrieval-augmented methods to enhance reliability in legal reasoning systems.

Abstract

Recent advances in test-time scaling of large language models (LLMs), exemplified by DeepSeek-R1 and OpenAI's o1, show that extending the chain of thought during inference can significantly improve general reasoning performance. However, the impact of this paradigm on legal reasoning remains insufficiently explored. To address this gap, we present the first systematic evaluation of 12 LLMs, including both reasoning-focused and general-purpose models, across 17 Chinese and English legal tasks spanning statutory and case-law traditions. In addition, we curate a bilingual chain-of-thought dataset for legal reasoning through distillation from DeepSeek-R1 and develop Legal-R1, an open-source model specialized for the legal domain. Experimental results show that Legal-R1 delivers competitive performance across diverse tasks. DeepSeek-R1 exhibits clear advantages in Chinese legal reasoning, while OpenAI's o1 achieves comparable results on English tasks. We further conduct a detailed error analysis, which reveals recurring issues such as outdated legal knowledge, limited capacity for legal interpretation, and susceptibility to factual hallucinations. These findings delineate the main obstacles confronting legal-domain LLMs and suggest promising directions for future research.

Figures (23)

• Figure 1: Overview of Work. The figure presents the 12 evaluated models together with representative task types and their data sources.
• Figure 2: Overall Performance of LLMs on Chinese and English Legal Tasks. The figure shows the performance of representative LLMs on Chinese and English legal tasks. Inference models such as DeepSeek-R1 and o1-preview outperform traditional LLMs, while our model Legal-R1 achieves competitive performance.
• Figure 3: Error types across typical legal tasks.
• Figure 4: The prompt for LC dataset.
• Figure 5: The prompt for LMHR dataset.