Can We Predict Before Executing Machine Learning Agents?

TL;DR

This work formalizes the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons, demonstrating that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report, achieving 61.5% accuracy and robust confidence calibration.

Abstract

Autonomous machine learning agents have revolutionized scientific discovery, yet they remain constrained by a Generate-Execute-Feedback paradigm. Previous approaches suffer from a severe Execution Bottleneck, as hypothesis evaluation relies strictly on expensive physical execution. To bypass these physical constraints, we internalize execution priors to substitute costly runtime checks with instantaneous predictive reasoning, drawing inspiration from World Models. In this work, we formalize the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons. We demonstrate that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report, achieving 61.5% accuracy and robust confidence calibration. Finally, we instantiate this framework in FOREAGENT, an agent that employs a Predict-then-Verify loop, achieving a 6x acceleration in convergence while surpassing execution-based baselines by +6%. Our code and dataset will be publicly available soon at https://github.com/zjunlp/predict-before-execute.

Figures (15)

• Figure 1: From Execution to Inference. Traditional ML agents improve through costly execution and external feedback, incurring substantial latency. Our work investigates whether superior data-grounded solutions can be identified before execution by leveraging "Implicit Execution Priors".
• Figure 2: Overview of the Framework.(a) Task Definition: The Data-centric Solution Preference task predicts solution superiority and confidence via latent reasoning. (b-c) Data Curation: We collect and filter real-world agent trajectories to construct the Preference Corpus. (d) Augmentation: Inputs are augmented with Verified Data Reports via a "Profile-Verify-Verbalize" pipeline. (e) ForeAgent Application: The model serves as a filter within the Predict-then-Verify loop, predicting preference before physical execution to prune candidates.
• Figure 3: Comprehensive Analysis of World Model Mechanisms and Capabilities.(a) Impact of Data Representation: Predictive success stems from semantic data understanding rather than complexity heuristics. (b) Domain Sensitivity: The superiority of verbal reports remains consistent across domains. (c) Scaling Laws: Accuracy decouples from pure parameter scaling. (d) Inference Dynamics: Active reasoning outperforms direct answering with robust stability across temperatures. (e) Calibration Analysis: Self-reported confidence strictly correlates with accuracy. (f) Complexity Discrimination: Accuracy scales with the complexity gap.
• Figure 4: Agent Performance Analysis. (a) Task-wise Beat Ratio:ForeAgent achieves an average +6% improvement over the AIDE baseline. (b) Temporal Efficiency: The agent converges to peak performance using only 1/6 of the execution time, achieving an average $6\times$ speedup. (c) Search Breadth: By offloading evaluation to the "Implicit World Model", ForeAgent explores $3.2\times$ more nodes on average compared to the baseline, significantly expanding the search space within the same time budget.
• Figure 5: Hierarchical distribution of the unique solution architectures in our Prediction Corpus. The chart illustrates the balance achieved across major machine learning paradigms: Gradient Boosting&Trees, General/Sequential NNs, CNNs, and Transformers. The outer ring details specific model instances, demonstrating the high heterogeneity of the solution space.