Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking

TL;DR

The paper addresses the leakage problem in backtesting LLM-driven trading by introducing DeepFund, a leakage-free live benchmark that evaluates LLMs using real-time market data post training cutoffs. It deploys a multi-agent framework (Financial Planner, Analyst Team, Portfolio Manager) within a live environment and a modular data/LLM interface to ensure fair, forward-looking evaluation. Nine state-of-the-art LLMs are tested on a live 24-day window focusing on Berkshire Hathaway's top holdings, with performance metrics such as $CR$, $CR_{bnh}$, $SR$, $MDD$, $WR$, $eta$, and $\alpha$ guiding assessment. The findings show substantial variability across models, with many incurring net losses and only a single model Grok 3 delivering profitability, underscoring current limits of LLMs for active fund management and demonstrating the need for leakage-free benchmarks in financial AI research.

Abstract

Large Language Models (LLMs) have demonstrated notable capabilities across financial tasks, including financial report summarization, earnings call transcript analysis, and asset classification. However, their real-world effectiveness in managing complex fund investment remains inadequately assessed. A fundamental limitation of existing benchmarks for evaluating LLM-driven trading strategies is their reliance on historical back-testing, inadvertently enabling LLMs to "time travel"-leveraging future information embedded in their training corpora, thus resulting in possible information leakage and overly optimistic performance estimates. To address this issue, we introduce DeepFund, a live fund benchmark tool designed to rigorously evaluate LLM in real-time market conditions. Utilizing a multi-agent architecture, DeepFund connects directly with real-time stock market data-specifically data published after each model pretraining cutoff-to ensure fair and leakage-free evaluations. Empirical tests on nine flagship LLMs from leading global institutions across multiple investment dimensions-including ticker-level analysis, investment decision-making, portfolio management, and risk control-reveal significant practical challenges. Notably, even cutting-edge models such as DeepSeek-V3 and Claude-3.7-Sonnet incur net trading losses within DeepFund real-time evaluation environment, underscoring the present limitations of LLMs for active fund management. Our code is available at https://github.com/HKUSTDial/DeepFund.

Figures (9)

• Figure 1: Shifts from static benchmarks to live benchmarks. Particularly for relevant date information in 1(a), we refer to public sources ( e.g., model card, arXiv, GitHub) for illustration.
• Figure 2: The DeepFund framework.
• Figure 3: Portfolio asset value for each LLM over time.
• Figure 4: AAPL trading for DeepSeek and Grok.
• Figure 5: Composition portfolio value for DeepSeek and Grok during the trading period. It shows the holding value of each ticker in the portfolio and the remaining cash flow.