Vietnamese Legal Information Retrieval in Question-Answering System

TL;DR

The paper tackles reliable Vietnamese legal information retrieval within a QA framework by integrating retrieval-augmented generation with Vietnamese-specific data processing, similarity-based clustering, and an Active Retrieval re-ranking mechanism. It combines BM25 and dense-vector search with a normalized fusion strategy and an LLM-driven re-ranking stage to improve answer accuracy and document ordering. Experimental results show that data processing, ensemble retrieval, and the LLM re-ranker yield strong performance gains, achieving high recall on processed datasets. The work demonstrates practical gains for Vietnamese legal QA and provides a public codebase for replication.

Abstract

In the modern era of rapidly increasing data volumes, accurately retrieving and recommending relevant documents has become crucial in enhancing the reliability of Question Answering (QA) systems. Recently, Retrieval Augmented Generation (RAG) has gained significant recognition for enhancing the capabilities of large language models (LLMs) by mitigating hallucination issues in QA systems, which is particularly beneficial in the legal domain. Various methods, such as semantic search using dense vector embeddings or a combination of multiple techniques to improve results before feeding them to LLMs, have been proposed. However, these methods often fall short when applied to the Vietnamese language due to several challenges, namely inefficient Vietnamese data processing leading to excessive token length or overly simplistic ensemble techniques that lead to instability and limited improvement. Moreover, a critical issue often overlooked is the ordering of final relevant documents which are used as reference to ensure the accuracy of the answers provided by LLMs. In this report, we introduce our three main modifications taken to address these challenges. First, we explore various practical approaches to data processing to overcome the limitations of the embedding model. Additionally, we enhance Reciprocal Rank Fusion by normalizing order to combine results from keyword and vector searches effectively. We also meticulously re-rank the source pieces of information used by LLMs with Active Retrieval to improve user experience when refining the information generated. In our opinion, this technique can also be considered as a new re-ranking method that might be used in place of the traditional cross encoder. Finally, we integrate these techniques into a comprehensive QA system, significantly improving its performance and reliability

Figures (3)

• Figure 1: Example of the data preprocessing process, when a large article is split into various overlapping chunks
• Figure 2: Architecture of our Information Retrieval System as well as Question Answer System
• Figure 3: An example of the Query rewriting module. Notice that the query might contain typos, abbreviations, and vagueness.