Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinCall-Surprise: A Large Scale Multi-modal Benchmark for Earning Surprise Prediction

Dong Shu, Yanguang Liu, Huopu Zhang, Mengnan Du

TL;DR

FinCall-Surprise introduces the first large-scale open-source multi-modal benchmark for earnings surprise prediction, combining synchronized conference-call transcripts, audio, and slides from 2,688 calls (2019–2021). The authors establish a comprehensive benchmark over 26 models, revealing that high accuracy can be illusory due to real-world class imbalance and that multimodal signals yield only modest, inconsistent gains with current architectures. They also find that some closed-source models show more robust, balanced predictions, while financial-domain fine-tuning can degrade basic instruction-following and generation abilities. By releasing this dataset and benchmark, the work aims to reduce reliance on proprietary data and spur development of more capable multimodal financial reasoning systems.

Abstract

Predicting corporate earnings surprises is a profitable yet challenging task, as accurate forecasts can inform significant investment decisions. However, progress in this domain has been constrained by a reliance on expensive, proprietary, and text-only data, limiting the development of advanced models. To address this gap, we introduce \textbf{FinCall-Surprise} (Financial Conference Call for Earning Surprise Prediction), the first large-scale, open-source, and multi-modal dataset for earnings surprise prediction. Comprising 2,688 unique corporate conference calls from 2019 to 2021, our dataset features word-to-word conference call textual transcripts, full audio recordings, and corresponding presentation slides. We establish a comprehensive benchmark by evaluating 26 state-of-the-art unimodal and multi-modal LLMs. Our findings reveal that (1) while many models achieve high accuracy, this performance is often an illusion caused by significant class imbalance in the real-world data. (2) Some specialized financial models demonstrate unexpected weaknesses in instruction-following and language generation. (3) Although incorporating audio and visual modalities provides some performance gains, current models still struggle to leverage these signals effectively. These results highlight critical limitations in the financial reasoning capabilities of existing LLMs and establish a challenging new baseline for future research.

FinCall-Surprise: A Large Scale Multi-modal Benchmark for Earning Surprise Prediction

TL;DR

FinCall-Surprise introduces the first large-scale open-source multi-modal benchmark for earnings surprise prediction, combining synchronized conference-call transcripts, audio, and slides from 2,688 calls (2019–2021). The authors establish a comprehensive benchmark over 26 models, revealing that high accuracy can be illusory due to real-world class imbalance and that multimodal signals yield only modest, inconsistent gains with current architectures. They also find that some closed-source models show more robust, balanced predictions, while financial-domain fine-tuning can degrade basic instruction-following and generation abilities. By releasing this dataset and benchmark, the work aims to reduce reliance on proprietary data and spur development of more capable multimodal financial reasoning systems.

Abstract

Predicting corporate earnings surprises is a profitable yet challenging task, as accurate forecasts can inform significant investment decisions. However, progress in this domain has been constrained by a reliance on expensive, proprietary, and text-only data, limiting the development of advanced models. To address this gap, we introduce \textbf{FinCall-Surprise} (Financial Conference Call for Earning Surprise Prediction), the first large-scale, open-source, and multi-modal dataset for earnings surprise prediction. Comprising 2,688 unique corporate conference calls from 2019 to 2021, our dataset features word-to-word conference call textual transcripts, full audio recordings, and corresponding presentation slides. We establish a comprehensive benchmark by evaluating 26 state-of-the-art unimodal and multi-modal LLMs. Our findings reveal that (1) while many models achieve high accuracy, this performance is often an illusion caused by significant class imbalance in the real-world data. (2) Some specialized financial models demonstrate unexpected weaknesses in instruction-following and language generation. (3) Although incorporating audio and visual modalities provides some performance gains, current models still struggle to leverage these signals effectively. These results highlight critical limitations in the financial reasoning capabilities of existing LLMs and establish a challenging new baseline for future research.

Paper Structure

This paper contains 36 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Earning Surprise Prediction Models
  4. Earning Surprise Prediction Datasets
  5. FinCall-Surprise Construction
  6. Data Construction Pipeline
  7. Earning Surprise Label Preliminaries
  8. Data Analysis
  9. Benchmark Setting
  10. Baseline LLMs
  11. Benchmark Input Design
  12. Text-Only Modality.
  13. Image-Text Modality.
  14. Audio-Text Modality.
  15. Benchmark Metrics
  16. ...and 21 more sections

Figures (2)

  • Figure 1: Overview of our data construction pipeline, which consists of two stages: (a) Firm Collection (left): We select large, publicly traded US firms based on market capitalization ($> \$1$B) and daily trading volume ($> \$50$M). (b) Data Collection (right): For each firm, we gather and synchronize three modalities for each quarterly earnings call: textual transcripts from Seeking Alpha, audio recordings from EarningsCast, and presentation slides from sources like Bloomberg Business.
  • Figure 2: Illustration of our summarization pipeline. (a) A data example showing a conference call transcript with speaker turns (Operator, Executives, Analysts) and its corresponding earning surprise label. (b) The summarization pipeline, where transcripts are grouped by speaker and iteratively summarized by the LLM until the total token count falls below the predefined limit.