Answer Retrieval in Legal Community Question Answering
Arian Askari, Zihui Yang, Zhaochun Ren, Suzan Verberne
TL;DR
This paper tackles answer retrieval in legal community question answering, addressing the knowledge gap between lawyers and non-experts and the mix of informal and formal content on legal QA sites. It introduces CEFS, a cross-encoder re-ranker that uses a fine-grained structured input (Subject, Description, Tags) and token splitter separators, trained on a new LegalQA benchmark (9,846 questions, 33,670 answers). The results show CEFS significantly outperforms MS MARCO-tuned cross-encoder baselines and a LegalQA-tuned CE re-ranker, with MAP@1k reaching 0.270 and strong ablation results highlighting the importance of Description and Tags. The LegalQA dataset and CEFS offer a path to applying this approach to other CQA domains where question tags are available, and the dataset enables future work such as legal response generation.
Abstract
The task of answer retrieval in the legal domain aims to help users to seek relevant legal advice from massive amounts of professional responses. Two main challenges hinder applying existing answer retrieval approaches in other domains to the legal domain: (1) a huge knowledge gap between lawyers and non-professionals; and (2) a mix of informal and formal content on legal QA websites. To tackle these challenges, we propose CE_FS, a novel cross-encoder (CE) re-ranker based on the fine-grained structured inputs. CE_FS uses additional structured information in the CQA data to improve the effectiveness of cross-encoder re-rankers. Furthermore, we propose LegalQA: a real-world benchmark dataset for evaluating answer retrieval in the legal domain. Experiments conducted on LegalQA show that our proposed method significantly outperforms strong cross-encoder re-rankers fine-tuned on MS MARCO. Our novel finding is that adding the question tags of each question besides the question description and title into the input of cross-encoder re-rankers structurally boosts the rankers' effectiveness. While we study our proposed method in the legal domain, we believe that our method can be applied in similar applications in other domains.