FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures

TL;DR

FigEx2 tackles the challenge of grounding and captioning in compound scientific figures by introducing a visual-conditioned architecture that localizes labeled panels and generates panel-specific captions from the figure alone. The method connects captioning to detection via a [DET] trigger and a gated fusion module that softly injects caption features into detector queries, stabilized through a four-stage training regime including SCST with CLIP and BERTScore rewards. Key contributions include the gated fusion mechanism, a four-stage SFT+RL optimization schedule, and BioSci-Fig-Cap along with PhysSci-Fig-Cap-Test and ChemSci-Fig-Cap-Test benchmarks for rigorous cross-domain evaluation. Empirical results show state-of-the-art mAP for panel detection and superior captioning metrics across in-domain and cross-domain datasets, with notable zero-shot transferability to physics and chemistry domains. The work demonstrates practical impact by enabling reliable panel-level understanding in scientific figures, even when captions are missing or noisy, and it highlights the benefit of few-shot prompts for rapid adaptation across disciplines.

Abstract

Scientific compound figures combine multiple labeled panels into a single image, but captions in real pipelines are often missing or only provide figure-level summaries, making panel-level understanding difficult. In this paper, we propose FigEx2, visual-conditioned framework that localizes panels and generates panel-wise captions directly from the compound figure. To mitigate the impact of diverse phrasing in open-ended captioning, we introduce a noise-aware gated fusion module that adaptively filters token-level features to stabilize the detection query space. Furthermore, we employ a staged optimization strategy combining supervised learning with reinforcement learning (RL), utilizing CLIP-based alignment and BERTScore-based semantic rewards to enforce strict multimodal consistency. To support high-quality supervision, we curate BioSci-Fig-Cap, a refined benchmark for panel-level grounding, alongside cross-disciplinary test suites in physics and chemistry. Experimental results demonstrate that FigEx2 achieves a superior 0.726 mAP@0.5:0.95 for detection and significantly outperforms Qwen3-VL-8B by 0.51 in METEOR and 0.24 in BERTScore. Notably, FigEx2 exhibits remarkable zero-shot transferability to out-of-distribution scientific domains without any fine-tuning.

Figures (6)

• Figure 1: Task overview of FigEx2. Input is only the compound figure. FigEx2 detects labeled panels and generates panel-wise captions. We highlight two practical challenges: figures with an overall caption only and figures with missing captions. The output is a set of labeled panel boxes paired with corresponding panel captions (bottom).
• Figure 2: Overall framework of FigEx2, containing two branches: Detection (top) and Captioning (bottom). Given an input compound figure, the captioning branch first generates structured panel-wise captions and outputs a [DET] token; the detector is then conditioned on the final-layer hidden state $h_{\mathrm{det}}$ to predict panel boxes and labels. FigEx2 is trained with supervised caption/detection objectives and an SCST-style reward-augmented stage using CLIP-based panel-caption alignment computed from ground-truth crops and BERTScore-based semantic rewards.
• Figure 3: Gated fusion module for conditioning detector queries on caption features with cross-attention and gated modulation. B denotes batch size.
• Figure 4: Compound figure with panel boxes with aligned subcaptions.
• Figure 5: Compound figure with panel boxes with aligned subcaptions.