A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences

TL;DR

This work addresses the challenge of transparent law reasoning by introducing a hierarchical, tree-structured schema that links evidences, factum probanda of varying granularity, and human experiences to support judicial decision-making. It formalizes Transparent Law Reasoning with Tree-Organized Structures (TL) and defines four sub-tasks to reconstruct a case's ultimate probandum from textual input, including evidence extraction/linking and experience generation, all evaluated via a crowd-sourced dataset built from publicly available judgments. An agent framework (TL Agent) with a dedicated toolkit and a ReAct-like strategy orchestrates multi-tool reasoning to produce structured, audit-friendly outputs, outperforming multiple baselines and showing the value of knowledge tools and multi-role reasoning. The dataset, methodology, and agent design advance transparent, accountable AI-assisted law reasoning with potential implications for public trust in judicial processes and future AI-enabled legal workflows.

Abstract

While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the ``Intelligent Court''.

Figures (13)

• Figure 1: Case "Rex v. Bywaters and Thompson" that demonstrates different experiences have impacted different results (LEFT vs. RIGHT). The case description and evidence are shared, but the experiences of both sides are different, which leads to different ultimate probandum.
• Figure 2: Illustration of the schema.
• Figure 3: Illustration of the task. For convenience, we showcase examples for each sub-task. The output of the 3 sub-tasks is collected to form the complete law reasoning structure.
• Figure 4: Illustration of the factum probandum generation.
• Figure 5: Illustration of the evidence extraction in subtask 2.