Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives

Jiaxin Liu, Yi Yang, Kar Yan Tam

TL;DR

The paper addresses the challenge of detecting subtle semantic shifts in financial narratives where surface similarity masks meaningful changes across periods. It defines a Financial-STS task and builds FinSTS, a two-part dataset (LLM-augmented and human-annotated) derived from Dow Jones 30 annual reports, to benchmark semantic-shift detection. An LLM-augmented data-generation pipeline produces quadruple-category examples, and a Triplet network trained on this data outperforms classic STS baselines and generic LLM embeddings, with notable gains on both augmented and human-annotated evaluation sets. The work provides a practical approach and public resources to improve market-relevant interpretation of longitudinal financial reporting, enabling more accurate assessment of a company's evolving financial and operational stance.

Abstract

In this paper, we introduce the Financial-STS task, a financial domain-specific NLP task designed to measure the nuanced semantic similarity between pairs of financial narratives. These narratives originate from the financial statements of the same company but correspond to different periods, such as year-over-year comparisons. Measuring the subtle semantic differences between these paired narratives enables market stakeholders to gauge changes over time in the company's financial and operational situations, which is critical for financial decision-making. We find that existing pretrained embedding models and LLM embeddings fall short in discerning these subtle financial narrative shifts. To address this gap, we propose an LLM-augmented pipeline specifically designed for the Financial-STS task. Evaluation on a human-annotated dataset demonstrates that our proposed method outperforms existing methods trained on classic STS tasks and generic LLM embeddings.

Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives

TL;DR

The paper addresses the challenge of detecting subtle semantic shifts in financial narratives where surface similarity masks meaningful changes across periods. It defines a Financial-STS task and builds FinSTS, a two-part dataset (LLM-augmented and human-annotated) derived from Dow Jones 30 annual reports, to benchmark semantic-shift detection. An LLM-augmented data-generation pipeline produces quadruple-category examples, and a Triplet network trained on this data outperforms classic STS baselines and generic LLM embeddings, with notable gains on both augmented and human-annotated evaluation sets. The work provides a practical approach and public resources to improve market-relevant interpretation of longitudinal financial reporting, enabling more accurate assessment of a company's evolving financial and operational stance.

Abstract

In this paper, we introduce the Financial-STS task, a financial domain-specific NLP task designed to measure the nuanced semantic similarity between pairs of financial narratives. These narratives originate from the financial statements of the same company but correspond to different periods, such as year-over-year comparisons. Measuring the subtle semantic differences between these paired narratives enables market stakeholders to gauge changes over time in the company's financial and operational situations, which is critical for financial decision-making. We find that existing pretrained embedding models and LLM embeddings fall short in discerning these subtle financial narrative shifts. To address this gap, we propose an LLM-augmented pipeline specifically designed for the Financial-STS task. Evaluation on a human-annotated dataset demonstrates that our proposed method outperforms existing methods trained on classic STS tasks and generic LLM embeddings.
Paper Structure (18 sections, 1 equation, 3 figures, 6 tables)

This paper contains 18 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. STS in Financial Narratives
  3. Proposed Pipeline
  4. Financial Semantic Shift Data Augmentation with LLM
  5. Triplet network for Measuring Financial Narrative Similarity
  6. The FinSTS dataset
  7. LLM-augmented FinSTS Dataset
  8. Dataset Assessment
  9. Human-annotated FinSTS Dataset
  10. Evaluation of Financial-STS task
  11. Financial-STS baselines
  12. Evaluation Dataset and Metrics
  13. Experiment results
  14. Related work
  15. Conclusion
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Cosine similarity between 4,027 paired financial statements encoded by OpenAI's Ada embedding ('text-ada-embedding-002') and SentenceBERT ('all-MiniLM-L6-v2'). The pairs were obtained from the annual reports of the Dow Jones Index component companies from year 2018 to 2019.
  • Figure 2: We propose to prompt LLM to generate financial narrative pairs that exhibit either no semantic shift or minimal shift, based on the identified semantic shift categories.
  • Figure 3: Annotation instructions.