LEXam: Benchmarking Legal Reasoning on 340 Law Exams

TL;DR

LEXam introduces a multilingual, long-form legal reasoning benchmark derived from 340 Swiss law exams, covering 78 subdomains with 2,841 open-ended questions and 1,660 multiple-choice questions across English and German. It couples open-ended prompt-based evaluation with an ensemble LLM-as-a-Judge validated against human experts via an Alternative Annotator Test, demonstrating alignment with human judgments and enabling scalable assessment beyond simple accuracy. Empirical results show state-of-the-art reasoning models excel at structured, multi-step legal reasoning, yet open-ended questions remain substantially challenging; MCQs reveal sensitivity to distractor quantity and language, highlighting robustness concerns. The work provides open-source data and evaluation pipelines, establishes expert-backed evaluation as a scalable tool for legal reasoning research, and outlines plans to broaden jurisdictional coverage and data collection.

Abstract

Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce \textsc{LEXam}, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. We have open-sourced our code on https://github.com/LEXam-Benchmark/LEXam and released our data on https://huggingface.co/datasets/LEXam-Benchmark/LEXam. Project page: https://lexam-benchmark.github.io.

Figures (14)

• Figure 1: Process for creating LEXam, a comprehensive legal reasoning benchmark derived from real law school exams. Created through careful expert extraction and curation, LEXam contains 2,841 open-ended and 2,045 multiple-choice questions (MCQs), each with detailed domain metadata. Open-ended questions support both process- and outcome-based evaluation by LLMs-as-a-Judge and human judges, while MCQs provide clear, outcome-based assessments.
• Figure 2: Illustration of a long-form open question (left, abbreviated example for illustration purposes. The full version is provided in Appendix \ref{['appendix:sample_lfq']}) and a MCQ with a set of candidate statements (right).
• Figure 3: Distribution of open questions and MCQs by legal area, language, and jurisdiction across development and test datasets. Open questions (solid) and MCQs (hatched).
• Figure 4: Model performance on open questions grouped by metadata.
• Figure 5: Model performance on MCQs grouped by various metadata.