Bilingual BSARD: Extending Statutory Article Retrieval to Dutch

TL;DR

The paper introduces bBSARD, a bilingual extension of BSARD that pairs Dutch and French Belgian statutory articles with Dutch translations of French questions, enabling robust cross-language retrieval benchmarking. It conducts extensive experiments across lexical, zero-shot dense, and fine-tuned models in both languages, finding BM25 remains a strong baseline while small language-specific models, when fine-tuned, can surpass zero-shot open models. The study highlights language disparities, with French generally easier to retrieve than Dutch, and demonstrates that bilingual or cross-language fine-tuning can improve Dutch performance. The dataset and code are made publicly available to foster further development of Dutch legal retrieval systems and cross-lingual capabilities in multilingual jurisdictions.

Abstract

Statutory article retrieval plays a crucial role in making legal information more accessible to both laypeople and legal professionals. Multilingual countries like Belgium present unique challenges for retrieval models due to the need for handling legal issues in multiple languages. Building on the Belgian Statutory Article Retrieval Dataset (BSARD) in French, we introduce the bilingual version of this dataset, bBSARD. The dataset contains parallel Belgian statutory articles in both French and Dutch, along with legal questions from BSARD and their Dutch translation. Using bBSARD, we conduct extensive benchmarking of retrieval models available for Dutch and French. Our benchmarking setup includes lexical models, zero-shot dense models, and fine-tuned small foundation models. Our experiments show that BM25 remains a competitive baseline compared to many zero-shot dense models in both languages. We also observe that while proprietary models outperform open alternatives in the zero-shot setting, they can be matched or surpassed by fine-tuning small language-specific models. Our dataset and evaluation code are publicly available.

Figures (3)

• Figure 1: Distribution of different codes in the bBSARD article corpus. 'Relevant' articles (green) are the ones cited in the question set. Light and dark blue columns correspond to the Federal and Regional codes, respectively.
• Figure 2: Basic statistics of bBSARD. From the left: Number of words in the articles (French and Dutch), Number of words in the questions (French and Dutch), number of relevant articles per question, and number of citations per relevant article.
• Figure 3: Standard Siamese Bi-Encoder setting with in-batch negatives, which we use for fine-tuning. Articles and Questions are encoded separately with the same model into vectors. For each question $Q_{i}$, the relevant article $A_{i}$ is the positive sample, while all other articles in the batch are used as negatives. $\odot$ represents the cosine similarity operator.