FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

TL;DR

FutureX introduces the first live, contamination-free benchmark for evaluating LLM agents on future prediction across 11 domains with daily automated question generation and ground-truth resolution. The framework combines a large-scale, semi-automated data pipeline with a diverse model zoo (base LLMs, reasoning/search agents, and deep research agents) and robust evaluation metrics that accommodate open-ended forecasting under uncertainty. Key findings show that tool-augmented reasoning boosts performance on hard, dynamic tasks, while strong base models excel on simpler, closed-form questions; humans still outperform agents on many tasks. The work demonstrates FutureX’s potential to drive progress toward professional-human-analyst-level forecasting in high-stakes domains through a scalable, real-time benchmarking platform.

Abstract

Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.

Figures (17)

• Figure 1: Overall scores on FutureX between July 20$^\text{th}$ and August 3$^\text{rd}$.
• Figure 2: The overall pipeline of FutureX, which consists of event database construction, future event daily curation, agent daily prediction, and answer daily acquisition. The entire pipeline is fully automated (except the event database construction which needs human efforts) and operates on a daily basis.
• Figure 3: The detailed future event daily curation process, which manipulates daily and weekly events from the event database. As an example, the number of events between July 24$^{\text{th}}$ and July 31$^{\text{st}}$ remaining after each step is shown in red. Note that the number of events each week varies due to fluctuations in prediction market events (the first row in the figure).
• Figure 4: Domains of all events included in FutureX, from July 20$^\text{th}$ to August 3$^\text{rd}$, total 1,272 events. These events are organized into 11 main categories---politics, sports, crypto, culture, finance, business, technology, weather, health, and space---with each category containing several sub-categories.
• Figure 5: Event type distribution (between July 20$^\text{th}$ and August 3$^\text{rd}$).