GPTs and Language Barrier: A Cross-Lingual Legal QA Examination
Ha-Thanh Nguyen, Hiroaki Yamada, Ken Satoh
TL;DR
The paper tackles cross-lingual legal question answering using GPT-3.5 and GPT-4 on the COLIEE Task 4 entailment task, examining English and Japanese prompts and contexts. It evaluates four prompting configurations (EN-EN, JA-JA, EN-JA, JA-EN) across multiple years, highlighting how language and translation affect legal reasoning in a multilingual setting. Results show GPT-4 consistently surpasses GPT-3.5, with monolingual Japanese prompts achieving the strongest performance, while cross-lingual configurations lag behind and remain challenging. The work underscores the need for high-quality translations and domain-specific knowledge to realize robust multilingual legal QA systems in practice.
Abstract
In this paper, we explore the application of Generative Pre-trained Transformers (GPTs) in cross-lingual legal Question-Answering (QA) systems using the COLIEE Task 4 dataset. In the COLIEE Task 4, given a statement and a set of related legal articles that serve as context, the objective is to determine whether the statement is legally valid, i.e., if it can be inferred from the provided contextual articles or not, which is also known as an entailment task. By benchmarking four different combinations of English and Japanese prompts and data, we provide valuable insights into GPTs' performance in multilingual legal QA scenarios, contributing to the development of more efficient and accurate cross-lingual QA solutions in the legal domain.