Top-Down Semantic Refinement for Image Captioning

TL;DR

The paper tackles the myopic generation problem in vision-language image captioning by reframing captioning as a goal-driven planning task. It introduces Top-Down Semantic Refinement (TDSR), a highly efficient MCTS-based planner with visual-guided parallel expansion, a lightweight value network, and adaptive early stopping to enable coarse-to-fine semantic refinement. Validated on DetailCaps, COMPOSITIONCAP, and POPE benchmarks across LLaVA-1.5 and Qwen2.5-VL backbones, TDSR achieves state-of-the-art or strongly competitive results in fine-grained description, compositional generalization, and hallucination suppression while maintaining reasonable latency. The approach proposes a general, plug-and-play planning framework for captioning that improves global coherence and detail without prohibitive compute costs, offering a principled alternative to bottom-up or purely autoregressive methods.

Abstract

Large Vision-Language Models (VLMs) face an inherent contradiction in image captioning: their powerful single-step generation capabilities often lead to a myopic decision-making process. This makes it difficult to maintain global narrative coherence while capturing rich details, a limitation that is particularly pronounced in tasks that require multi-step and complex scene description. To overcome this fundamental challenge, we redefine image captioning as a goal-oriented hierarchical refinement planning problem, and further propose a novel framework, named Top-Down Semantic Refinement (TDSR), which models the generation process as a Markov Decision Process (MDP). However, planning within the vast state space of a VLM presents a significant computational hurdle. Our core contribution, therefore, is the design of a highly efficient Monte Carlo Tree Search (MCTS) algorithm tailored for VLMs. By incorporating a visual-guided parallel expansion and a lightweight value network, our TDSR reduces the call frequency to the expensive VLM by an order of magnitude without sacrificing planning quality. Furthermore, an adaptive early stopping mechanism dynamically matches computational overhead to the image's complexity. Extensive experiments on multiple benchmarks, including DetailCaps, COMPOSITIONCAP, and POPE, demonstrate that our TDSR, as a plug-and-play module, can significantly enhance the performance of existing VLMs (e.g., LLaVA-1.5, Qwen2.5-VL) by achieving state-of-the-art or highly competitive results in fine-grained description, compositional generalization, and hallucination suppression.

Figures (9)

• Figure 1: The TDSR framework generates coherent, detailed captions through global-to-local refinement, guided by redundancy-aware stopping and efficient MCTS.
• Figure 2: The architecture of TDSR's MCTS planner. The four canonical stages, i.e., selection, visual-guided parallel expansion, lightweight value estimation, and backpropagation, are tailored to efficiently search within VLMs. Composite rewards combine local precision and global coherence.
• Figure 3: POPE benchmark results. TDSR improves hallucination robustness across random, popular, and adversarial settings.
• Figure 4: Efficiency-performance tradeoff of TDSR. The full framework achieves the best generation quality (BERTScore) with only a marginal latency increase, clearly outperforming prior methods.
• Figure 5: Step-wise CIDEr score under ablation settings. Removing any core component from TDSR results in significant performance drops, especially in early stopping and value-guided region selection.