Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering

TL;DR

This work addresses legal question answering by proposing a three-stage framework, PFR-LQA, that combines domain-specific pre-training, task-specific fine-tuning, and contextual re-ranking to enhance dense retrieval with a dual-encoder. It employs self-supervised and context-supervised pre-training to tailor representations to legal text ($L_{\rm pretrain}$), followed by fine-tuning with cosine similarity and the circle loss $\mathcal{L}_{2}$, including hard negatives. The re-ranking stage uses BM25-guided query aggregation and a Transformer to optimize a joint objective $\mathcal{L}_{\rm joint}$ comprised of $\mathcal{L}_{\rm CL}$ and $\mathcal{L}_{\rm MSE}$, with refined affinity $Z^{q}$. Evaluated on the LawQA Chinese legal QA dataset, PFR-LQA substantially surpasses baselines in $P@1$ and $\text{MRR@16}$, demonstrating the value of integrating domain knowledge, supervision, and contextual similarity for legal QA.

Abstract

Legal question answering (QA) has attracted increasing attention from people seeking legal advice, which aims to retrieve the most applicable answers from a large-scale database of question-answer pairs. Previous methods mainly use a dual-encoder architecture to learn dense representations of both questions and answers. However, these methods could suffer from lacking domain knowledge and sufficient labeled training data. In this paper, we propose a three-stage (\underline{p}re-training, \underline{f}ine-tuning and \underline{r}e-ranking) framework for \underline{l}egal \underline{QA} (called PFR-LQA), which promotes the fine-grained text representation learning and boosts the performance of dense retrieval with the dual-encoder architecture. Concretely, we first conduct domain-specific pre-training on legal questions and answers through a self-supervised training objective, allowing the pre-trained model to be adapted to the legal domain. Then, we perform task-specific fine-tuning of the dual-encoder on legal question-answer pairs by using the supervised learning objective, leading to a high-quality dual-encoder for the specific downstream QA task. Finally, we employ a contextual re-ranking objective to further refine the output representations of questions produced by the document encoder, which uses contextual similarity to increase the discrepancy between the anchor and hard negative samples for better question re-ranking. We conduct extensive experiments on a manually annotated legal QA dataset. Experimental results show that our PFR-LQA method achieves better performance than the strong competitors for legal question answering.

Figures (1)

• Figure 1: The training framework of contextual re-ranking.