Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection

TL;DR

Spec-o3 tackles the bottleneck of expert vetting in rare celestial object catalogs by introducing a tool-augmented vision-language agent that inspects spectra through interleaved multimodal reasoning. It combines a two-stage post-training pipeline—cold-start supervised fine-tuning on expert inspection trajectories followed by outcome-based reinforcement learning—with a spectral visualization tool to emulate astronomer workflows. The approach achieves state-of-the-art macro-F1 on five LAMOST-based rare-object tasks, with strong cross-survey and cross-task generalization and coherent, expert-aligned reasoning traces. This work offers a scalable, interpretable framework for spectral vetting that can adapt to the data deluge in upcoming spectroscopic surveys and provides a reproducible benchmark (SpecVI-Bench).

Abstract

Due to the limited generalization and interpretability of deep learning classifiers, The final vetting of rare celestial object candidates still relies on expert visual inspection--a manually intensive process. In this process, astronomers leverage specialized tools to analyze spectra and construct reliable catalogs. However, this practice has become the primary bottleneck, as it is fundamentally incapable of scaling with the data deluge from modern spectroscopic surveys. To bridge this gap, we propose Spec-o3, a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection via interleaved multimodal chain-of-thought reasoning. Spec-o3 is trained with a two-stage post-training recipe: cold-start supervised fine-tuning on expert inspection trajectories followed by outcome-based reinforcement learning on rare-type verification tasks. Evaluated on five rare-object identification tasks from LAMOST, Spec-o3 establishes a new State-of-the-Art, boosting the macro-F1 score from 28.3 to 76.5 with a 7B parameter base model and outperforming both proprietary VLMs and specialized deep models. Crucially, the agent demonstrates strong generalization to unseen inspection tasks across survey shifts (from LAMOST to SDSS/DESI). Expert evaluations confirm that its reasoning traces are coherent and physically consistent, supporting transparent and trustworthy decision-making. Code, data, and models are available at \href{https://github.com/Maxwell-Jia/spec-o3}{Project HomePage}.

Figures (7)

• Figure 1: (a) Astronomers' visual inspection workflow. Astronomers visualize the raw numerical arrays to assess global morphology, then iteratively zoom into specific wavelength regions to examine fine-grained features for the final decision. (b) Performance comparison. Spec-o3 achieves state-of-the-art performance and good generalization across all datasets.
• Figure 2: An illustration of Spec-o3's Interleaved Multimodal Chain-of-Thought. The agent iteratively alternates between textual reasoning (<think>…</think>) and fine-grained visual evidence from tool-rendered zoomed spectrum plots. Red JSON shows the tool calls. The final decision is in <answer>...</answer>.
• Figure 3: Overview of the Spec-o3 framework. Given a prompt $T_0$ and an initial view $I_0$, the VLM generates an iMCoT trajectory in which text reasoning blocks $T_n$ are interleaved with tool-generated images parameterized by wavelength interval $\Delta\lambda$ and optional label $l_n$, until the final text output $T_N$ is produced. The VLM is cold-start initialized and optimized with GRPO.
• Figure 4: (a) Quality distribution. Score distributions and Cumulative Distribution Functions are compared between human and LLM judges to evaluate reasoning trajectories. (b) Rating consistency. The heatmap displays Spearman correlation coefficients between human experts and four LLM judges. (c) Pairwise preference. Human expert preferences on explanation quality are compared between Spec-o3 and o3 across three datasets.
• Figure 5: Screenshot of the official LAMOST spectrum viewer. The left panel lists object metadata, the center panel visualizes the spectrum, and the right panel provides interactive controls. The red box highlights the wavelength-range selection tool for zooming and re-rendering spectral details. Left: global spectrum view. Right: zoomed view re-rendered over 6400-6700 Å.