Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Massive Legal Embedding Benchmark (MLEB)

Umar Butler, Abdur-Rahman Butler, Adrian Lucas Malec

TL;DR

The Massive Legal Embedding Benchmark (MLEB) addresses the inadequacy of existing legal information retrieval benchmarks by introducing a large, diverse, open benchmark that spans six jurisdictions, five document types, and three task types. It comprises ten evaluation sets, seven of which are newly created to broaden domain and jurisdictional coverage, and emphasizes high-quality labeling and real-world utility for legal RAG scenarios. The paper analyzes model performance, highlighting that domain-adapted, law-focused embeddings (e.g., Kanon 2 Embedder, Voyage variants) outperform general-purpose models on MLEB, and discusses speed considerations and limitations such as licensing constraints and potential data leakage. By releasing data and code openly, MLEB enables reproducible evaluations and sets the stage for ongoing expansion to better support practitioners and researchers in legal information retrieval.

Abstract

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.

The Massive Legal Embedding Benchmark (MLEB)

TL;DR

The Massive Legal Embedding Benchmark (MLEB) addresses the inadequacy of existing legal information retrieval benchmarks by introducing a large, diverse, open benchmark that spans six jurisdictions, five document types, and three task types. It comprises ten evaluation sets, seven of which are newly created to broaden domain and jurisdictional coverage, and emphasizes high-quality labeling and real-world utility for legal RAG scenarios. The paper analyzes model performance, highlighting that domain-adapted, law-focused embeddings (e.g., Kanon 2 Embedder, Voyage variants) outperform general-purpose models on MLEB, and discusses speed considerations and limitations such as licensing constraints and potential data leakage. By releasing data and code openly, MLEB enables reproducible evaluations and sets the stage for ongoing expansion to better support practitioners and researchers in legal information retrieval.

Abstract

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.
Paper Structure (23 sections, 2 figures, 2 tables)

This paper contains 23 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. LegalBench-RAG
  4. MTEB-Legal
  5. The Massive Legal Embedding Benchmark (MLEB)
  6. Bar Exam QA
  7. SCALR
  8. Singaporean Judicial Keywords
  9. GDPR Holdings Retrieval
  10. Australian Tax Guidance Retrieval
  11. Irish Legislative Summaries
  12. UK Legislative Long Titles
  13. Contractual Clause Retrieval
  14. License TL;DR Retrieval
  15. Consumer Contracts QA
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Performance of the top 10 embedding models on MLEB, broken down by domain type. Scores represent NDCG@10 averaged across all datasets within each domain (Judicial, Contractual, Regulatory). Legal domain-adapted models (Kanon 2 Embedder, Voyage 3 Large, Voyage 3.5, Voyage Law 2) show particularly strong performance.
  • Figure 2: NDCG@10 score versus total evaluation time for commercial embedding models on MLEB. Evaluation time includes network latency and reflects real-world conditions (batch size of 16 for documents, 1 for queries). The plot demonstrates the speed-accuracy tradeoff among commercial offerings.