Integrating Domain Knowledge for Financial QA: A Multi-Retriever RAG Approach with LLMs

TL;DR

The paper tackles financial QA tasks that require precise domain knowledge and multi-step numerical reasoning. It introduces a multi-retriever RAG framework that combines internal context retrieval with external finance definitions, and supports both neural-symbolic and prompt-based LLM generators. Domain-specific training with SecBERT boosts neural-symbolic performance beyond the FinQA baseline, while a prompt-based LLM achieves state-of-the-art improvements (>7%), though still below expert humans; the study also highlights a trade-off between hallucinations and external knowledge gains, with larger models benefiting more from external facts and few-shot prompts. The results demonstrate improved numerical reasoning capabilities in the latest LLMs when tailored with few-shot learning and external knowledge retrieval.

Abstract

This research project addresses the errors of financial numerical reasoning Question Answering (QA) tasks due to the lack of domain knowledge in finance. Despite recent advances in Large Language Models (LLMs), financial numerical questions remain challenging because they require specific domain knowledge in finance and complex multi-step numeric reasoning. We implement a multi-retriever Retrieval Augmented Generators (RAG) system to retrieve both external domain knowledge and internal question contexts, and utilize the latest LLM to tackle these tasks. Through comprehensive ablation experiments and error analysis, we find that domain-specific training with the SecBERT encoder significantly contributes to our best neural symbolic model surpassing the FinQA paper's top model, which serves as our baseline. This suggests the potential superior performance of domain-specific training. Furthermore, our best prompt-based LLM generator achieves the state-of-the-art (SOTA) performance with significant improvement (>7%), yet it is still below the human expert performance. This study highlights the trade-off between hallucinations loss and external knowledge gains in smaller models and few-shot examples. For larger models, the gains from external facts typically outweigh the hallucination loss. Finally, our findings confirm the enhanced numerical reasoning capabilities of the latest LLM, optimized for few-shot learning.

Figures (3)

• Figure 1: Sample question and answer from the FinQA dataset
• Figure 2: Model's architecture
• Figure 3: Training Loss Over Rounds (20 epochs) for the Generator with SEC-BERT encoder and LSTM decoder.