Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages

Sayan Mahapatra, Debtanu Datta, Shubham Soni, Adrijit Goswami, Saptarshi Ghosh

TL;DR

This work constructs the first high-quality legal parallel corpus containing aligned text units in English and nine Indian languages, that includes several low-resource languages and benchmark the performance of a wide variety of Machine Translation systems over this corpus.

Abstract

Most legal text in the Indian judiciary is written in complex English due to historical reasons. However, only a small fraction of the Indian population is comfortable in reading English. Hence legal text needs to be made available in various Indian languages, possibly by translating the available legal text from English. Though there has been a lot of research on translation to and between Indian languages, to our knowledge, there has not been much prior work on such translation in the legal domain. In this work, we construct the first high-quality legal parallel corpus containing aligned text units in English and nine Indian languages, that includes several low-resource languages. We also benchmark the performance of a wide variety of Machine Translation (MT) systems over this corpus, including commercial MT systems, open-source MT systems and Large Language Models. Through a comprehensive survey by Law practitioners, we check how satisfied they are with the translations by some of these MT systems, and how well automatic MT evaluation metrics agree with the opinions of Law practitioners.

MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages

TL;DR

This work constructs the first high-quality legal parallel corpus containing aligned text units in English and nine Indian languages, that includes several low-resource languages and benchmark the performance of a wide variety of Machine Translation systems over this corpus.

Abstract

Most legal text in the Indian judiciary is written in complex English due to historical reasons. However, only a small fraction of the Indian population is comfortable in reading English. Hence legal text needs to be made available in various Indian languages, possibly by translating the available legal text from English. Though there has been a lot of research on translation to and between Indian languages, to our knowledge, there has not been much prior work on such translation in the legal domain. In this work, we construct the first high-quality legal parallel corpus containing aligned text units in English and nine Indian languages, that includes several low-resource languages. We also benchmark the performance of a wide variety of Machine Translation (MT) systems over this corpus, including commercial MT systems, open-source MT systems and Large Language Models. Through a comprehensive survey by Law practitioners, we check how satisfied they are with the translations by some of these MT systems, and how well automatic MT evaluation metrics agree with the opinions of Law practitioners.
Paper Structure (23 sections, 2 equations, 5 figures, 14 tables)

This paper contains 23 sections, 2 equations, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The MILPaC Datasets
  4. Dataset 1: MILPaC-IP
  5. Dataset 2: MILPaC-CCI-FAQ
  6. Dataset 3: MILPaC-Acts
  7. Comparing the three datasets in MILPaC
  8. Quality of the datasets
  9. Availability of the corpus
  10. Benchmarking Translation Systems
  11. Translation systems
  12. Metrics for automatic evaluation
  13. Results of automatic evaluation
  14. Evaluation through Law practitioners
  15. Analysis of errors committed by the MT systems
  16. ...and 8 more sections

Figures (5)

  • Figure 1: A flow chart describing our methodology on benchmarking MT systems for legal text translation in Indian languages.
  • Figure 2: Examples of parallel text pairs from the 3 datasets in MILPaC. From each dataset, we show one textual unit in all available languages in that dataset.
  • Figure 3: Examples of errors committed by various MT systems across the MILPaC datasets. The examples are selected from among the textual units that were used in the surveys by the Law students and obtained low scores. The $3^{rd}$ column states the target language (in which the translated text unit in the $2^{nd}$ column is), and the $4^{th}$ column states the MT system. The last column gives the expert scores in the range $[0, 5]$ (the metrics POM, SLU, FLY are explained in the text) and a remark/justification given by the expert.
  • Figure 4: Example of text written in an old Bengali font which is often confusing for modern OCRs
  • Figure 5: Example of erased / eroded characters, probably due to poor quality scans. OCR has no way of giving correct output for these erased characters.