AInstein: Assessing the Feasibility of AI-Generated Approaches to Research Problems

TL;DR

AInstein systematically tests whether large language models can solve AI research problems using only their parametric knowledge, by decoupling problem extraction from solution generation and enforcing iterative critique loops that mimic scientific review. The framework is evaluated on 1,214 ICLR 2025 abstracts, using an LLM-as-judge to score problem quality via a deficit metric $d$ and solution quality via a $Feasibility,Completeness,Novelty$ rubric, plus embedding- and human-verified ELO rankings. Results show that the internal model capability is the dominant predictor of success (e.g., internal GPT-OSS-120B achieves high SRs, while external pairings matter less), but Rediscovery dramatically drops under strict evaluation, whereas Novel & Valid proposals remain strong, illustrating a mix of genuine reasoning and creative problem-solving with framing sensitivity. The study provides large-scale evidence of latent reasoning in LLMs, highlights current limitations in robust autonomous problem solving, and introduces a rigorous methodological paradigm for future cross-domain evaluation of AI-driven discovery.

Abstract

Large language models (LLMs) demonstrate impressive capabilities across a wide range of tasks, yet it remains unclear whether such success reflects genuine reasoning or sophisticated recall. We introduce AInstein, a framework for testing whether LLMs can generate valid solutions to AI research problems using only their pretrained parametric knowledge -- without domain-specific fine-tuning, retrieval augmentation, or other external aids. Our approach extracts distilled problem statements from high-quality ICLR 2025 submissions, then tasks specialized solver agents with proposing and refining technical solutions through iterative critique loops, mimicking the cycles of proposal, review, and revision central to scientific inquiry. We evaluate AInstein on 1,214 ICLR papers stratified by acceptance tier (Oral, Spotlight, Poster), using an LLM-as-a-judge paradigm guided by a structured rubric, complemented by targeted manual checks. Performance is assessed with three metrics: Success Rate (does the solution address the problem?), Rediscovery (does it align with human-proposed methods?), and Novelty (does it yield valid, original approaches?). Our results reveal that while LLMs can rediscover feasible solutions and occasionally propose creative alternatives, their problem-solving ability remains fragile and highly sensitive to framing. These findings provide the first large-scale evidence on the extent to which LLMs can act as autonomous scientific problem-solvers, highlighting both their latent potential and their current limitations.

Figures (4)

• Figure 1: The AInstein framework. An input scientific abstract ($\mathcal{A}$) is first derived into a generalized problem ($\mathcal{P}$) by the Generalizer agent ($\mathcal{G}$). The Solver agent ($\mathcal{S}$) then attempts to derive a technical solution ($\mathcal{Z}$), using the problem statement $\mathcal{P}$. Both phases employ an iterative refinement loop with internal ($\mathcal{M}_i$) and external ($\mathcal{M}_e$) critique. Note: The transition $\mathcal{A} \to \mathcal{P}$ follows the same iterative refinement mechanism, where the the "Happy" condition is as described in Section \ref{['sec:refinement']}.
• Figure 2: Correlation matrix for Generalizer quality metrics. The deficit score ($d$) is strongly correlated with Information Loss and Ambiguity.
• Figure 3: Performance comparison of internal models across ICLR paper tiers (Oral, Spotlight, Poster), averaging across GPT-OSS-120B as the external model. The results are shown for the strict threshold ($\tau=5$). The performance hierarchy is stable, and the top model (GPT-OSS-120B) is not significantly impacted by the paper's prestige.
• Figure 4: Visual analysis of the 11 identified research clusters. (a) Top keywords provide a thematic summary for several clusters. (b) The UMAP projection illustrates the distinct grouping of solutions, with each color corresponding to a unique thematic cluster.