CoT-PL: Visual Chain-of-Thought Reasoning Meets Pseudo-Labeling for Open-Vocabulary Object Detection

TL;DR

This work tackles open-vocabulary object detection by addressing weaknesses in single-step image-text alignment-based pseudo-labeling, especially in crowded or occluded scenes. It introduces CoT-PL, a unified pipeline that decomposes object understanding into region perception with SAM, zero-shot category recognition via multimodal large language models, and background grounding, augmented by a Contrastive Background Learning objective to disentangle foreground from background. The approach yields high-quality pseudo-labels, achieving state-of-the-art results on OV-COCO and OV-LVIS, and demonstrates the benefit of visual chain-of-thought reasoning combined with background-aware contrastive learning. The findings indicate that stronger MLLMs lead to proportional gains, and that background grounding effectively mitigates background collapse, with practical implications for robust open-set detection in complex visual scenes.

Abstract

Open-vocabulary object detection (OVD) seeks to recognize and localize object categories beyond those seen during training. Recent approaches typically leverage vision-language models (VLMs) to generate pseudo-labels using image-text alignment, allowing detectors to generalize to unseen classes without explicit supervision. However, these methods depend heavily on direct image-text matching, neglecting the intermediate reasoning steps essential for interpreting semantically complex scenes. This results in limited robustness when confronted with crowded or occluded visual contexts. In this paper, we introduce CoT-PL, a new framework that employs structured visual chain-of-thought (CoT) reasoning into the pseudo-labeling process. CoT-PL decomposes object understanding into three interpretable steps: (1) region perception even for unseen objects, (2) category recognition via zero-shot reasoning, and (3) background grounding to separate semantically complex objects. Crucially, the third step naturally motivates our contrastive background learning (CBL) that uses the pre-computed background cues as negatives to promote feature disentanglement between objects and background. In this way, CoT reasoning and CBL form an integrated pipeline tailored to robust pseudo-labeling in crowded or occluded scenes. Notably, in these two settings, our novel-class pseudo-label quality achieves relative improvements of 103.4% and 168.4% over the best prior, respectively. Our extensive experiments demonstrate that CoT-PL achieves +7.7 AP50 on open-vocabulary COCO and +2.9 mask AP on LVIS for novel classes, setting a new state of the art. Code and models are available at https://github.com/hchoi256/cotpl.

Figures (13)

• Figure 1: Trends in pseudo-labeling for OVD.(a) Manual pseudo-labels for novel classes are costly and do not scale. (b) Recent self-training methods automate pseudo-labeling by labeling region proposals via similarity with category text embeddings using vision-language models (VLMs), but degrade with VLMs’ poor object localization and caption-dependent vocabulary. (c) We employ visual chain-of-thought with Segment Anything Model SAM and multimodal large language models qwen for accurate object perception and zero-shot category recognition.
• Figure 2: Limitations and quality of pseudo-labels for complex scenes.(a) Noisy pseudo boxes due to poor object localization by VLMs, (b) limited object coverage from captions, and (c) occluded objects treated as background. (d) Pseudo-label quality on the OV-COCO validation set: Crowded denotes images with many objects and Occluded denotes objects occluded by other objects.
• Figure 3: Overview of the proposed visual chain-of-thought pipeline. Our method first queries an MLLM about object existence inside SAM-generated pseudo boxes on preprocessed images, then performs zero-shot labeling, and finally extracts background representations nearby. Refined with semantic anchors for reliability, the resulting pseudo-annotations are merged into the base set.
• Figure 4: Overall architecture of CoT-PL. Built on BARON baron (See Appendix \ref{['appendix:baseline']}), the proposed method encodes the open-world base set, partially including novel classes, using the CLIP text encoder. The CLIP embeddings of multiple background concepts are averaged to initialize a single learnable background embedding. These concepts are also used as negative samples in contrastive learning to encourage feature disentanglement between objects and background. At inference time, we apply CBL++ to mitigate class interference by removing pseudo-labels associated with the ground-truth novel classes.
• Figure 5: The overall architecture of BARON baron. To reduce sampling time during training, BARON’s naive neighbor sampling can optionally be replaced with our semantic anchor-based strategy. In particular, caching the anchors reduced training time by 25% compared to the baseline, while maintaining the original performance.