AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinement

TL;DR

AstroReview tackles the bottleneck of competitive telescope time by delivering an open-source, LLM-driven multi-agent framework that splits proposal evaluation into novelty, feasibility, and meta-review stages, with mechanisms for memory and reliability verification. The system demonstrates that a staged, task-isolated approach reduces hallucinations and yields trustworthy assessments, achieving $87.08\%$ accuracy in the final decision stage and driving a $66\%$ improvement in acceptance for revised drafts in iterative cycles. Key contributions include a three-stage evaluation pipeline, a closed-loop review-refine workflow with a Proposal Authoring Agent, a robust Review Agent ensemble, and a reliability verifier that augments output quality. The work points to a practical path for scalable, auditable, and higher-throughput proposal review at resource-limited facilities, with potential extension to full-text proposals and observation-simulation integration to further enhance feasibility and scientific yield.

Abstract

Competitive access to modern observatories has intensified as proposal volumes outpace available telescope time, making timely, consistent, and transparent peer review a critical bottleneck for the advancement of astronomy. Automating parts of this process is therefore both scientifically significant and operationally necessary to ensure fair allocation and reproducible decisions at scale. We present AstroReview, an open-source, agent-based framework that automates proposal review in three stages: (i) novelty and scientific merit, (ii) feasibility and expected yield, and (iii) meta-review and reliability verification. Task isolation and explicit reasoning traces curb hallucinations and improve transparency. Without any domain specific fine tuning, AstroReview used in our experiments only for the last stage, correctly identifies genuinely accepted proposals with an accuracy of 87%. The AstroReview in Action module replicates the review and refinement loop; with its integrated Proposal Authoring Agent, the acceptance rate of revised drafts increases by 66% after two iterations, showing that iterative feedback combined with automated meta-review and reliability verification delivers measurable quality gains. Together, these results point to a practical path toward scalable, auditable, and higher throughput proposal review for resource limited facilities.

Figures (4)

• Figure 1: Overview of AstroReview. The framework emulates human expert evaluation across three key dimensions: novelty and scientific merit, feasibility and expected yield, and meta-review and reliability verification.
• Figure 2: Workflow of the iterative proposal review and refinement. The process starts with user inputs that specify scientific targets or phenomena and the observing instrument (Step 0). In Step 1 (Proposal and Refinement), the Proposal Authoring Agent drafts or revises the proposal using prior feedback stored in the Manuscript Memory. In Step 2 (Review), multiple Review Agents generate scores and comments, which are aggregated by the Meta-Review Agent and then audited by the Reliability Verifier for template compliance, logical coherence, and evidence alignment; the resulting records are stored in the Review Memory. In Step 3 (Decision Gate), the system decides whether to stop or continue another iteration based on the proposal content and reviewer feedback. The loop repeats until the stopping condition is satisfied.
• Figure 3: Evolution of proposal evaluation metrics across three refinement rounds. Violin plots depicting the full distribution of scores in each round (left); Bar chart summarizing the acceptance and rejection rate achieved in each round (right).
• Figure 4: Representative Review Agent outputs generated using Qwen‑2.5‑72B and Llama‑3.3-70B back‑ends.