Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Agentic Autonomous Scientific Discovery with Vision-Language Model Capabilities

Kahaan Gandhi, Boris Bolliet, Inigo Zubeldia

TL;DR

The paper presents a vision-language model–driven, multi-agent framework that uses plots as verifiable checkpoints to self-correct and steer autonomous scientific discovery. By evaluating figures against dynamically generated domain rubrics, the Plot Judge and related agents enable real-time debugging and exploratory decision-making without human input. Empirical demonstrations in cosmology and astrochemistry show that VLM feedback yields substantial gains on a 10-task discovery benchmark (pass@1 ≈ 0.7–0.8) compared with text-only baselines (≈0.2–0.5) and provides auditable reasoning traces for interpretability. The work argues for generalizable, interpretable automation in data-intensive science while acknowledging the need for safeguards in deployment.

Abstract

We show that multi-agent systems guided by vision-language models (VLMs) improve end-to-end autonomous scientific discovery. By treating plots as verifiable checkpoints, a VLM-as-a-judge evaluates figures against dynamically generated domain-specific rubrics, enabling agents to correct their own errors and steer exploratory data analysis in real-time. Case studies in cosmology and astrochemistry demonstrate recovery from faulty reasoning paths and adaptation to new datasets without human intervention. On a 10-task benchmark for data-driven discovery, VLM-augmented systems achieve pass at 1 scores of 0.7-0.8, compared to 0.2-0.3 for code-only and 0.4-0.5 for code-and-text baselines, while also providing auditable reasoning traces that improve interpretability. Code available here: https://github.com/CMBAgents/cmbagent

Enhancing Agentic Autonomous Scientific Discovery with Vision-Language Model Capabilities

TL;DR

The paper presents a vision-language model–driven, multi-agent framework that uses plots as verifiable checkpoints to self-correct and steer autonomous scientific discovery. By evaluating figures against dynamically generated domain rubrics, the Plot Judge and related agents enable real-time debugging and exploratory decision-making without human input. Empirical demonstrations in cosmology and astrochemistry show that VLM feedback yields substantial gains on a 10-task discovery benchmark (pass@1 ≈ 0.7–0.8) compared with text-only baselines (≈0.2–0.5) and provides auditable reasoning traces for interpretability. The work argues for generalizable, interpretable automation in data-intensive science while acknowledging the need for safeguards in deployment.

Abstract

We show that multi-agent systems guided by vision-language models (VLMs) improve end-to-end autonomous scientific discovery. By treating plots as verifiable checkpoints, a VLM-as-a-judge evaluates figures against dynamically generated domain-specific rubrics, enabling agents to correct their own errors and steer exploratory data analysis in real-time. Case studies in cosmology and astrochemistry demonstrate recovery from faulty reasoning paths and adaptation to new datasets without human intervention. On a 10-task benchmark for data-driven discovery, VLM-augmented systems achieve pass at 1 scores of 0.7-0.8, compared to 0.2-0.3 for code-only and 0.4-0.5 for code-and-text baselines, while also providing auditable reasoning traces that improve interpretability. Code available here: https://github.com/CMBAgents/cmbagent

Paper Structure

This paper contains 13 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. System Overview
  5. Automating Self-Correction with VLM Feedback
  6. Steering Discovery Systems with VLM Feedback
  7. Results
  8. Case Study: Correcting CMB Power Spectra
  9. Case Study: Exploring Spectral Line Models
  10. Evaluation on Scientific Discovery Tasks
  11. Conclusion
  12. Benchmark Tasks
  13. Prompt Templates

Figures (2)

  • Figure 1: Self-correction workflow with a VLM-as-a-judge. Center: feedback loop where plots are evaluated by a VLM against a domain-specific rubric, routed to the Plot Judge and Plot Debugger agents when criteria are not met, and iteratively revised. Left: Pydantic schemas defining structured outputs, with icons showing where each schema is applied. Right: first attempt by coding agents to plot the lensed CMB TT power spectrum from Sec. \ref{['sec:CMB']}, where $D_\ell^{TT}$ was incorrectly rescaled, and the corrected version after a single pass, now in agreement with Planck 2018 $\Lambda$CDM predictions.
  • Figure 2: Extending scientific workflows with VLM-guided exploration. Top right: when plots reveal features warranting further investigation, the system initiates exploratory data analysis to adapt the research trajectory and update beliefs with intermediate findings. Top left: Pydantic schemas define structured outputs for experiment design and evaluation, in addition to those shown in Fig. \ref{['fig:correction']}. Bottom: three phases of the case study in Sec. \ref{['sec:CaseStudy']} — (1) fitting the null hypothesis of a single Gaussian, (2) testing alternative spectral line models, and (3) selecting the self-absorption model, correctly inferring the true distribution of the data.