Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt

Peter Anderson, Mano Vikash Janardhanan, Jason He, Wei Cheng, Charlie Flanagan

TL;DR

BAM embeddings are presented, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries.

Abstract

Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. Yet, few text embeddings specialized for finance have been reported in the literature, perhaps in part due to a lack of public datasets and benchmarks. We present BAM embeddings, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs. Demonstrating the benefits of domain-specific training, BAM embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embedding from OpenAI. Further, BAM embeddings increase question answering accuracy by 8% on FinanceBench and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries. To support further research we describe our approach in detail, quantify the importance of hard negative mining and dataset scale.

Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt

TL;DR

BAM embeddings are presented, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries.

Abstract

Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. Yet, few text embeddings specialized for finance have been reported in the literature, perhaps in part due to a lack of public datasets and benchmarks. We present BAM embeddings, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs. Demonstrating the benefits of domain-specific training, BAM embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embedding from OpenAI. Further, BAM embeddings increase question answering accuracy by 8% on FinanceBench and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries. To support further research we describe our approach in detail, quantify the importance of hard negative mining and dataset scale.

Paper Structure

This paper contains 31 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Financial LLMs
  4. Domain-Specific Text Embeddings
  5. Dataset
  6. Sampling Text Passages
  7. Raw Documents
  8. Parsing and Splitting
  9. Document Context
  10. Query Generation
  11. Few-Shot Examples
  12. LLM Query Generation
  13. Filtering
  14. Query Realism
  15. BAM Embeddings
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Passage retrieval results: Recall@1 on a held-out test split of 447K query-passage pairs. BAM embeddings finetuned for financial document retrieval significantly outperform general-purpose embeddings.
  • Figure 2: FinanceBench results under the Shared Vector Store setting. Replacing OpenAI ada-002 embeddings with BAM embeddings increases accuracy by 8%.
  • Figure 3: Comparison of vector search using BAM embeddings with lexical search (BM25). Vector search is superior to lexical search, and improves on longer and more detailed queries (while lexical search degrades).