FinBERT-QA: Financial Question Answering with pre-trained BERT Language Models

TL;DR

The paper tackles financial non-factoid QA by framing it as a re-ranking problem using a two-stage pipeline: an Answer Retriever based on BM25 and an Answer Re-ranker built from pre-trained BERT variants. The authors propose FinBERT-QA, a Transfer-and-Adapt fine-tuned model that first trains on a large general QA corpus (MS MACRO) and then adapts to the FiQA dataset, achieving state-of-the-art gains on FiQA task 2. Key findings show that pointwise fine-tuning is generally more efficient and that Transfer and Adapt provides stronger performance than further pre-training; overall FinBERT-QA yields significant improvements in MRR and NDCG and competitive Precision@1. The work demonstrates the practical viability of domain-adapted transformer models for financial QA and highlights the trade-offs between retrieval quality, sequence length limits, and data quality, suggesting directions for future improvements and datasets.

Abstract

Motivated by the emerging demand in the financial industry for the automatic analysis of unstructured and structured data at scale, Question Answering (QA) systems can provide lucrative and competitive advantages to companies by facilitating the decision making of financial advisers. Consequently, we propose a novel financial QA system using the transformer-based pre-trained BERT language model to address the limitations of data scarcity and language specificity in the financial domain. Our system focuses on financial non-factoid answer selection, which retrieves a set of passage-level texts and selects the most relevant as the answer. To increase efficiency, we formulate the answer selection task as a re-ranking problem, in which our system consists of an Answer Retriever using BM25, a simple information retrieval approach, to first return a list of candidate answers, and an Answer Re-ranker built with variants of pre-trained BERT language models to re-rank and select the most relevant answers. We investigate various learning, further pre-training, and fine-tuning approaches for BERT. Our experiments suggest that FinBERT-QA, a model built from applying the Transfer and Adapt further fine-tuning and pointwise learning approach, is the most effective, improving the state-of-the-art results of task 2 of the FiQA dataset by 16% on MRR, 17% on NDCG, and 21% on Precision@1.

Figures (16)

• Figure 1: Overview of the QA pipeline. The Anserini Answer Retriever first returns the top 50 candidate answers. The Answer Re-ranker is built by (1) transferring a pre-trained BERT model to the large-scale MS MACRO dataset and (2) adapting the transfered BERT model to the target FiQA dataset. The Answer Re-ranker then outputs the top 10 most relevant answers.
• Figure 2: A feedforward network with one hidden and one output layer where $x_{1}...x_{n_{0}}$ is a set of $n$-dimensional inputs, $h_{1}...h_{n_{1}}$ is the output of the hidden layer, and $y_{1}...y_{n_{2}}$ is the result of the output layer. $W$ is the weight matrix of the hidden layer, $U$ is the weight matrix of the output layer, and $b$ is the bias vector jurafsky.
• Figure 3: RNN-based encoder-decoder architecture where the encoder passes the final hidden state $h_{n}$ of an RNN into the decoder. The decoder then autoregressively generates a task-specific novel output sequence jurafsky.
• Figure 4: Stacked encoders and decoders of the Transformer architecture for a French to English translation task alammar.
• Figure 5: The Transformer model architecture where the left block is the encoder and the right is the decoder. Both the encoder and decoder contain multi-head attention, normalization, and fully-connected feed-forward sublayers. The decoder also contains the masked multi-head attention layer transformers.