ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

TL;DR

ArabLegalEval introduces a multitask Arabic legal knowledge benchmark for large language models by combining native Saudi regulatory content with translations of English LegalBench tasks. It synthesizes MCQs, NajizQA open-ended questions, and translated contracts/privacy tasks, and evaluates them using both generation-time prompting strategies and retrieval augmented setups. The paper demonstrates how in-context learning, prompt optimization, and translator choice affect model performance, with GPT-4 and Arabic-centric models like Jais showing strong capabilities on Arabic legal reasoning while identifying tasks that remain challenging. By releasing the dataset and methodology, the work aims to accelerate Arabic legal AI research and enable domain-specific benchmarking across languages and jurisdictions.

Abstract

The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs' legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset's quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: https://github.com/Thiqah/ArabLegalEval

Figures (37)

• Figure 1: Tasks included in ArabLegalEval and their source documents.
• Figure 2: MCQs Generation and Filtering
• Figure 3: GPT-4 vs Claude-3-opus MCQs Generation
• Figure 4: NajizQA curation pipeline
• Figure 5: DSPy's prompt optimizer process.