Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models

TL;DR

The paper formalizes Deep Data Research (DDR) as an open-ended, autonomous data-exploration task for large language models and introduces DDR-Bench, a scalable, checklist-based benchmark built on real-world databases to objectively verify model-generated insights. It demonstrates that frontier models show emergent investigatory agency but still struggle with long-horizon exploration, highlighting the importance of intrinsic strategies and trainer-focused development over mere scaling. DDR-Bench separates execution from evaluation to curb data leakage and uses per-sample checklists and LLM-based verification to ground insights in verifiable data. The findings suggest that progress hinges on robust planning-like exploration, adaptive termination, and principled training pipelines that cultivate end-to-end autonomous data discovery, with strong emphasis on trust, hallucination control, and societal considerations.

Abstract

The agency expected of Agentic Large Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We term this investigatory intelligence, distinguishing it from executional intelligence, which merely completes assigned tasks. Data Science provides a natural testbed, as real-world analysis starts from raw data rather than explicit queries, yet few benchmarks focus on it. To address this, we introduce Deep Data Research (DDR), an open-ended task where LLMs autonomously extract key insights from databases, and DDR-Bench, a large-scale, checklist-based benchmark that enables verifiable evaluation. Results show that while frontier models display emerging agency, long-horizon exploration remains challenging. Our analysis highlights that effective investigatory intelligence depends not only on agent scaffolding or merely scaling, but also on intrinsic strategies of agentic models.

Figures (25)

• Figure 2: Left: Compared with previous tasks, DDR maximises exploration openness and agency, focusing on the direct evaluation of insight quality. Right: Overview of the DDR-Bench. Details of the trajectory samples are shown in Appendix \ref{['appendix:traj_sample']}.
• Figure 3: A case of Claude Sonnet 4.5's trajectory and evaluation checklist in the MIMIC scenario of DDR-Bench. Verified fact and supporting insights are underlined. See details of this trajectory in Figure \ref{['fig:traj_sample_mimic']}. The patient id is anonymised.
• Figure 4: Ranking correlation between novelty and accuracy on Proprietary and Open-Source LLMs. Circles denote the novelty rank, and diamonds denote the accuracy rank. Models are ordered by accuracy rank in the figure. All three scenarios present high correlation.
• Figure 5: Exploration patterns of different models. The x-axis denotes exploration entropy, reflecting the depth of the model’s search over the database, while the y-axis represents database coverage, indicating the breadth of the search.
• Figure 6: Self-termination visualisation on the Qwen family.