Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALARB: An Arabic Legal Argument Reasoning Benchmark

Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, Sameer Alsabea, Omar Shaqaqi, Hebah AlShamlan, Omar Knio, George Turkiyyah

TL;DR

ALARB introduces a large, native Arabic legal reasoning dataset derived from Saudi commercial-court cases, linking facts, court reasoning, verdicts, and cited statutes. It defines two task families—verdict prediction and article identification—and demonstrates the utility of instruction tuning on a 12B model to reach near GPT-4o performance on verdict generation. The dataset enables open-ended, multistep legal reasoning in Arabic and highlights the benefits of reasoning-enhanced prompts and cross-language experimentation. Limitations include geographic scope and dataset size, but results indicate substantial potential for improving Arabic reasoning in specialized domains through targeted fine-tuning and RL methods. Overall, ALARB provides a practical resource for evaluating and advancing Arabic legal reasoning in LLMs.

Abstract

We introduce ALARB, a dataset and suite of tasks designed to evaluate the reasoning capabilities of large language models (LLMs) within the Arabic legal domain. While existing Arabic benchmarks cover some knowledge-intensive tasks such as retrieval and understanding, substantial datasets focusing specifically on multistep reasoning for Arabic LLMs, especially in open-ended contexts, are lacking. The dataset comprises over 13K commercial court cases from Saudi Arabia, with each case including the facts presented, the reasoning of the court, the verdict, as well as the cited clauses extracted from the regulatory documents. We define a set of challenging tasks leveraging this dataset and reflecting the complexity of real-world legal reasoning, including verdict prediction, completion of reasoning chains in multistep legal arguments, and identification of relevant regulations based on case facts. We benchmark a representative selection of current open and closed Arabic LLMs on these tasks and demonstrate the dataset's utility for instruction tuning. Notably, we show that instruction-tuning a modest 12B parameter model using ALARB significantly enhances its performance in verdict prediction and Arabic verdict generation, reaching a level comparable to that of GPT-4o.

ALARB: An Arabic Legal Argument Reasoning Benchmark

TL;DR

ALARB introduces a large, native Arabic legal reasoning dataset derived from Saudi commercial-court cases, linking facts, court reasoning, verdicts, and cited statutes. It defines two task families—verdict prediction and article identification—and demonstrates the utility of instruction tuning on a 12B model to reach near GPT-4o performance on verdict generation. The dataset enables open-ended, multistep legal reasoning in Arabic and highlights the benefits of reasoning-enhanced prompts and cross-language experimentation. Limitations include geographic scope and dataset size, but results indicate substantial potential for improving Arabic reasoning in specialized domains through targeted fine-tuning and RL methods. Overall, ALARB provides a practical resource for evaluating and advancing Arabic legal reasoning in LLMs.

Abstract

We introduce ALARB, a dataset and suite of tasks designed to evaluate the reasoning capabilities of large language models (LLMs) within the Arabic legal domain. While existing Arabic benchmarks cover some knowledge-intensive tasks such as retrieval and understanding, substantial datasets focusing specifically on multistep reasoning for Arabic LLMs, especially in open-ended contexts, are lacking. The dataset comprises over 13K commercial court cases from Saudi Arabia, with each case including the facts presented, the reasoning of the court, the verdict, as well as the cited clauses extracted from the regulatory documents. We define a set of challenging tasks leveraging this dataset and reflecting the complexity of real-world legal reasoning, including verdict prediction, completion of reasoning chains in multistep legal arguments, and identification of relevant regulations based on case facts. We benchmark a representative selection of current open and closed Arabic LLMs on these tasks and demonstrate the dataset's utility for instruction tuning. Notably, we show that instruction-tuning a modest 12B parameter model using ALARB significantly enhances its performance in verdict prediction and Arabic verdict generation, reaching a level comparable to that of GPT-4o.

Paper Structure

This paper contains 33 sections, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Arabic LLM benchmarks
  4. Legal reasoning benchmarks and tasks
  5. Dataset
  6. Data Curation
  7. Case and Statutes Scraping.
  8. Data Cleaning and Mapping to Regulations.
  9. Case Restructuring and Anonymization.
  10. Dataset statistics
  11. Benchmark Tasks
  12. Verdict Prediction Tasks
  13. Task 1: From Facts Only.
  14. Task 2: From Facts and Relevant Articles.
  15. Task 3: From Facts and Court's Reasoning.
  16. ...and 18 more sections

Figures (11)

  • Figure 1: ALARB includes a dataset of structured legal cases. Each case lists the facts presented by the plaintiff and defendant, and an explicit step-by-step chain of the argument reasoning of the court leading to a verdict. Cases are linked to individual articles of applicable statutes and regulations. A set of legal reasoning tasks leverages the data. ALARB is available https://huggingface.co/datasets/HarethahMo/ALARB.
  • Figure 2: Data Preparation Workflow.
  • Figure 3: Distributions of Words and Steps.
  • Figure 4: SFT Training: Example from the verdict prediction task.
  • Figure 5: Error rate with partial reasoning.
  • ...and 6 more figures