PriceSeer: Evaluating Large Language Models in Real-Time Stock Prediction

TL;DR

PriceSeer addresses the need for a live, contamination-free benchmark to evaluate LLMs in stock price forecasting by integrating time-series data with multi-modal external signals, including news and veracity cues, across $110$ stocks in $11$ sectors and horizons $h\in\{3,5,10\}$ days. It constructs a prompt-driven pipeline that combines historical prices $P_{s,t}$, features $I_{s,t}$, and news $N_{s,t}$ into context $\mathcal{C}_{s,t}$, producing forecasts $\hat{p}_{s,t+h}$ and subsequent investment strategies with an initial capital of $10{,}000$ USD. The study evaluates six cutting-edge LLMs, revealing horizon- and sector-dependent performance—DeepSeek-V3.2 excels in short-term forecasting, while GPT-5 often leads in longer horizons—alongside notable sensitivity to fake news, which primarily harms near-term accuracy but attenuates over longer horizons. Ablation analyses show external signals boost short-term accuracy but can hurt longer-term predictions, underscoring the need to understand information source reliability in finance-focused LLM applications. Overall, PriceSeer provides a publicly available, multi-modal benchmark that advances understanding of LLM robustness and strategy generation in real-time financial markets, with practical implications for risk-aware, horizon-aware decision-making in investment contexts.

Abstract

Stock prediction, a subject closely related to people's investment activities in fully dynamic and live environments, has been widely studied. Current large language models (LLMs) have shown remarkable potential in various domains, exhibiting expert-level performance through advanced reasoning and contextual understanding. In this paper, we introduce PriceSeer, a live, dynamic, and data-uncontaminated benchmark specifically designed for LLMs performing stock prediction tasks. Specifically, PriceSeer includes 110 U.S. stocks from 11 industrial sectors, with each containing 249 historical data points. Our benchmark implements both internal and external information expansion, where LLMs receive extra financial indicators, news, and fake news to perform stock price prediction. We evaluate six cutting-edge LLMs under different prediction horizons, demonstrating their potential in generating investment strategies after obtaining accurate price predictions for different sectors. Additionally, we provide analyses of LLMs' suboptimal performance in long-term predictions, including the vulnerability to fake news and specific industries. The code and evaluation data will be open-sourced at https://github.com/BobLiang2113/PriceSeer.

Figures (6)

• Figure 1: Overview of PriceSeer. We collected stock data in the form of time series and textual news data, covering 11 representative sectors and 249 historical trading days. Based on this, we implemented quantitative finance metric-based information augmentation and introduced three tampering ways to disturb price prediction. The tasks were also designed in multiple prediction horizons.
• Figure 2: The template prompt used in PriceSeer. The green and red are specific to the scenarios with financial indicators and news, respectively.
• Figure 3: Performance comparison between different sectors based on pure historical data.
• Figure 4: Pearson correlation between different sectors in short-term, medium-term, and long-term prediction horizons.
• Figure 5: The distribution of investment strategies. "PnL" denotes the profit and loss.