Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems

TL;DR

This work addresses the challenge of reliable financial question answering with Retrieval-Augmented Generation (RAG) by building an end-to-end pipeline tailored to finance. It advances three pillars: (1) domain-aware pre-retrieval preprocessing of queries and financial corpora (e.g., markdown restructuring), (2) domain-adapted retrieval via fine-tuned dense embeddings and a hybrid dense-sparse scoring framework with tunable alpha, and (3) post-retrieval grounding through reranking and document selection, complemented by DPO-trained generation. The approach yields substantial gains across seven finance QA datasets, with the fine-tuned Stella_1.5B retriever achieving top NDCG@10 scores and query/corpus preprocessing driving notable improvements; the generation component, using a DPO-based agent and a selection mechanism, further enhances answer relevance and grounding. The results demonstrate the practical impact of tailored retrieval for finance, and the authors provide a replicable pipeline on GitHub for broader adoption and future refinements.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a promising framework to mitigate hallucinations in Large Language Models (LLMs), yet its overall performance is dependent on the underlying retrieval system. In the finance domain, documents such as 10-K reports pose distinct challenges due to domain-specific vocabulary and multi-hierarchical tabular data. In this work, we introduce an efficient, end-to-end RAG pipeline that enhances retrieval for financial documents through a three-phase approach: pre-retrieval, retrieval, and post-retrieval. In the pre-retrieval phase, various query and corpus preprocessing techniques are employed to enrich input data. During the retrieval phase, we fine-tuned state-of-the-art (SOTA) embedding models with domain-specific knowledge and implemented a hybrid retrieval strategy that combines dense and sparse representations. Finally, the post-retrieval phase leverages Direct Preference Optimization (DPO) training and document selection methods to further refine the results. Evaluations on seven financial question answering datasets-FinDER, FinQABench, FinanceBench, TATQA, FinQA, ConvFinQA, and MultiHiertt-demonstrate substantial improvements in retrieval performance, leading to more accurate and contextually appropriate generation. These findings highlight the critical role of tailored retrieval techniques in advancing the effectiveness of RAG systems for financial applications. A fully replicable pipeline is available on GitHub: https://github.com/seohyunwoo-0407/GAR.

Figures (2)

• Figure 1: Our enhanced RAG pipeline
• Figure 2: $\alpha$-NDCG@10 Graphs by each task