Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptation of Embedding Models to Financial Filings via LLM Distillation

Eliot Brenner, Dominic Seyler, Manjunath Hegde, Andrei Simion, Koustuv Dasgupta, Bing Xiang

TL;DR

This work tackles the high cost and limited domain relevance of generic embedding models for financial filings by introducing a scalable, LLM-guided distillation pipeline that trains a compact bi-encoder retriever from unlabeled SEC filings for RAG. The core idea is an iterative teacher-student framework where an LLM generates synthetic queries and mines hard positives/negatives from the corpus, which are used to fine-tune the retriever via triplet loss across 14 filing types. The approach achieves substantial retrieval improvements (MRR@5 up 27.7%, mean DCG@5 up 44.6%) and shows strong FinanceBench NDCG gains, validating both in-domain and out-of-distribution performance. The paper also analyzes the benefits of positive-example mining and discusses limitations and avenues for future work, such as integrating external knowledge graphs and agentic retrieval strategies to further enhance domain-specific search in finance.

Abstract

Despite advances in generative large language models (LLMs), practical application of specialized conversational AI agents remains constrained by computation costs, latency requirements, and the need for precise domain-specific relevance measures. While existing embedding models address the first two constraints, they underperform on information retrieval in specialized domains like finance. This paper introduces a scalable pipeline that trains specialized models from an unlabeled corpus using a general purpose retrieval embedding model as foundation. Our method yields an average of 27.7% improvement in MRR$\texttt{@}$5, 44.6% improvement in mean DCG$\texttt{@}$5 across 14 financial filing types measured over 21,800 query-document pairs, and improved NDCG on 3 of 4 document classes in FinanceBench. We adapt retrieval embeddings (bi-encoder) for RAG, not LLM generators, using LLM-judged relevance to distill domain knowledge into a compact retriever. There are prior works which pair synthetically generated queries with real passages to directly fine-tune the retrieval model. Our pipeline differs from these by introducing interaction between student and teacher models that interleaves retrieval-based mining of hard positive/negative examples from the unlabeled corpus with iterative retraining of the student model's weights using these examples. Each retrieval iteration uses the refined student model to mine the corpus for progressively harder training examples for the subsequent training iteration. The methodology provides a cost-effective solution to bridging the gap between general-purpose models and specialized domains without requiring labor-intensive human annotation.

Adaptation of Embedding Models to Financial Filings via LLM Distillation

TL;DR

This work tackles the high cost and limited domain relevance of generic embedding models for financial filings by introducing a scalable, LLM-guided distillation pipeline that trains a compact bi-encoder retriever from unlabeled SEC filings for RAG. The core idea is an iterative teacher-student framework where an LLM generates synthetic queries and mines hard positives/negatives from the corpus, which are used to fine-tune the retriever via triplet loss across 14 filing types. The approach achieves substantial retrieval improvements (MRR@5 up 27.7%, mean DCG@5 up 44.6%) and shows strong FinanceBench NDCG gains, validating both in-domain and out-of-distribution performance. The paper also analyzes the benefits of positive-example mining and discusses limitations and avenues for future work, such as integrating external knowledge graphs and agentic retrieval strategies to further enhance domain-specific search in finance.

Abstract

Despite advances in generative large language models (LLMs), practical application of specialized conversational AI agents remains constrained by computation costs, latency requirements, and the need for precise domain-specific relevance measures. While existing embedding models address the first two constraints, they underperform on information retrieval in specialized domains like finance. This paper introduces a scalable pipeline that trains specialized models from an unlabeled corpus using a general purpose retrieval embedding model as foundation. Our method yields an average of 27.7% improvement in MRR5, 44.6% improvement in mean DCG5 across 14 financial filing types measured over 21,800 query-document pairs, and improved NDCG on 3 of 4 document classes in FinanceBench. We adapt retrieval embeddings (bi-encoder) for RAG, not LLM generators, using LLM-judged relevance to distill domain knowledge into a compact retriever. There are prior works which pair synthetically generated queries with real passages to directly fine-tune the retrieval model. Our pipeline differs from these by introducing interaction between student and teacher models that interleaves retrieval-based mining of hard positive/negative examples from the unlabeled corpus with iterative retraining of the student model's weights using these examples. Each retrieval iteration uses the refined student model to mine the corpus for progressively harder training examples for the subsequent training iteration. The methodology provides a cost-effective solution to bridging the gap between general-purpose models and specialized domains without requiring labor-intensive human annotation.

Paper Structure

This paper contains 20 sections, 20 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Method
  5. Scaffold Iterative Process
  6. Address Sparsity of Answers in Corpus
  7. Associate Document Class to Queries
  8. Corpus-wide Retrieval to Select $\,C_{q,t}$
  9. Improve Scalability to Large Corpora
  10. Random Sampling and Corpus Partition
  11. Downsampling within $\,C(D_{\rm tv}(q,K,i))$
  12. Generation of "Synthetic" Queries
  13. Final Evaluation Pipeline
  14. Scaffolding of Evaluation Pipeline
  15. Model selection
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Training/validation pipeline overview. The iterative components of the pipeline are subscripted with "i".
  • Figure 2: Final evaluation pipeline. The iterative components of the pipeline are subscripted with "i". The test corpus is fixed for all iterations.
  • Figure 3: Metrics with standard errors as measured on FinanceBench Evaluation Set
  • Figure 4: Plots of $\mathrm{t\hbox{-}SNE}\!\left(\mathcal{S}(q)\right)$, the low-dimensional projections of difference vectors of positive and negative chunks, with $\mathrm{t\hbox{-}SNE}\!\left(\mathcal{S}_{\mathrm{seed}}(q)\right)$ in black, highlighting the benefits of positive example mining in obtaining a more diverse training set.