OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand

TL;DR

OpenExempt addresses the challenge of evaluating legal reasoning by offering a dynamic benchmark built on structured, machine-readable representations of statutes. The framework enables on-demand generation of tasks across assets, jurisdictions, and complexity, with a deterministic solver producing objective gold solutions for ground-truth evaluation. The benchmark comprises nine suites (six diagnostic, three competency) and nearly 10k samples to enable fine-grained probing of reasoning skills and error propagation across 13 models. Empirical results reveal sharp performance cliffs on longer reasoning paths and under obfuscation, underscoring the need for diagnostic evaluation to guide the development of more capable legal reasoning systems.

Abstract

Reasoning benchmarks have played a crucial role in the progress of language models. Yet rigorous evaluation remains a significant challenge as static question-answer pairs provide only a snapshot of performance, compressing complex behavior into a single accuracy metric. This limitation is especially true in complex, rule-bound domains such as law, where existing benchmarks are costly to build and ill suited for isolating specific failure modes. To address this, we introduce OpenExempt, a framework and benchmark for diagnostic evaluation of legal reasoning. The OpenExempt Framework uses expert-crafted symbolic representations of U.S. Bankruptcy Code statutes to dynamically generate a large space of natural language reasoning tasks and their machine-computable solutions on demand. This gives users fine-grained control over task complexity and scope, allowing individual reasoning skills to be probed in isolation. Using this system, we construct the OpenExempt Benchmark, a diagnostic benchmark for legal reasoning with 9,765 samples across nine evaluation suites designed to carefully probe model capabilities. Experiments on 13 diverse language models reveal sharp performance cliffs that emerge only under longer reasoning paths and in the presence of obfuscating statements. We release the framework and benchmark publicly to support research aimed at understanding and improving the next generation of reasoning systems.

Figures (9)

• Figure 1: OpenExempt tasks center on U.S. bankruptcy law, primarily asset exemption where assets must be assigned to exemption statutes with dollar limits.
• Figure 2: OpenExempt Framework Architecture. Dynamic task generation driven by user-defined configuration and grounded in structured legal knowledge.
• Figure 3: OpenExempt Task Pipeline. Each task depends on the successful completion of its predecessors, forming a composable sequence in which users can select any slice to isolate evaluation of specific reasoning types.
• Figure 4: Sample distribution across benchmark suites.
• Figure 5: Example Task EC (Exemption Classification) prompt from the Distractor Robustness evaluation suite; response format instructions and statutes abridged for brevity.