Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence

Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, Erik Cambria

TL;DR

LexGenius presents an expert-level Chinese legal GI benchmark structured as Dimension–Task–Ability to diagnose multi-stage legal reasoning, ethical judgment, and societal impact. It builds 8,385 MCQs from scratch, rigorously reviewed to minimize leakage, and evaluates 12 SOTA LLMs against a human baseline, revealing substantial gaps in legal general intelligence, especially in soft intelligence and normative judgment. The framework enables precise diagnostic insight into where models fail and how enhancements like GRPO can improve performance, guiding targeted development toward professional-grade legal AI. The work also identifies key limitations and outlines a path for extending the benchmark to multimodal, cross-linguistic, and temporally aware evaluations to better reflect real-world legal practice.

Abstract

Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making, simulating the expertise of legal experts across domains. However, existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs), hindering the development of legal GI. To address this, we propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs. It follows a Dimension-Task-Ability framework, covering seven dimensions, eleven tasks, and twenty abilities. We use the recent legal cases and exam questions to create multiple-choice questions with a combination of manual and LLM reviews to reduce data leakage risks, ensuring accuracy and reliability through multiple rounds of checks. We evaluate 12 state-of-the-art LLMs using LexGenius and conduct an in-depth analysis. We find significant disparities across legal intelligence abilities for LLMs, with even the best LLMs lagging behind human legal professionals. We believe LexGenius can assess the legal intelligence abilities of LLMs and enhance legal GI development. Our project is available at https://github.com/QwenQKing/LexGenius.

LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence

TL;DR

LexGenius presents an expert-level Chinese legal GI benchmark structured as Dimension–Task–Ability to diagnose multi-stage legal reasoning, ethical judgment, and societal impact. It builds 8,385 MCQs from scratch, rigorously reviewed to minimize leakage, and evaluates 12 SOTA LLMs against a human baseline, revealing substantial gaps in legal general intelligence, especially in soft intelligence and normative judgment. The framework enables precise diagnostic insight into where models fail and how enhancements like GRPO can improve performance, guiding targeted development toward professional-grade legal AI. The work also identifies key limitations and outlines a path for extending the benchmark to multimodal, cross-linguistic, and temporally aware evaluations to better reflect real-world legal practice.

Abstract

Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making, simulating the expertise of legal experts across domains. However, existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs), hindering the development of legal GI. To address this, we propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs. It follows a Dimension-Task-Ability framework, covering seven dimensions, eleven tasks, and twenty abilities. We use the recent legal cases and exam questions to create multiple-choice questions with a combination of manual and LLM reviews to reduce data leakage risks, ensuring accuracy and reliability through multiple rounds of checks. We evaluate 12 state-of-the-art LLMs using LexGenius and conduct an in-depth analysis. We find significant disparities across legal intelligence abilities for LLMs, with even the best LLMs lagging behind human legal professionals. We believe LexGenius can assess the legal intelligence abilities of LLMs and enhance legal GI development. Our project is available at https://github.com/QwenQKing/LexGenius.

Paper Structure

This paper contains 32 sections, 30 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LexGenius Framework
  4. Dimension: Education and Career Drive
  5. Task: Theory and Practice Coexist
  6. Ability: Constructivist Learning-based
  7. LexGenius Construction
  8. Construction Principles
  9. Construction Workflow
  10. Data Statistics
  11. Evaluation Method
  12. Experiments and Results
  13. Experimental Setup
  14. Main Results (RQ1)
  15. Legal Soft Intelligence Analysis (RQ2)
  16. ...and 17 more sections

Figures (30)

  • Figure 1: Comparison of LLM and human legal experts shows that humans outperform LLMs in all dimensions.
  • Figure 2: LexGenius can be divided into 3 levels: The first level includes Dimensions 1-7, the second level includes Tasks 1-11, and the third level includes Abilities 1-20 (A. 1 to A. 20). Each is numbered for reference in the text.
  • Figure 3: The MCQ construction workflow of the LexGenius, which is a process where LLM and manual work are combined. It includes three steps: data collection and structuring, construction of MCQs, and manual review.
  • Figure 4: Data distribution of LexGenius. Left: the MCQ proportions across different laws and the dimensions, tasks, and abilities. Right: the MCQ proportions of abilities. The PRC: the People's Republic of China.
  • Figure 5: Comparison of the 12 SOTA LLMs with human experts on 7 core dimensions of legal intelligence.
  • ...and 25 more figures