CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction

TL;DR

CoT-Seg addresses robust vision-language segmentation under implicit and complex queries by integrating chain-of-thought reasoning with a self-correction loop in a training-free, multi-agent framework. An MLLM Reasoner crafts stepwise meta-queries, a Segmentation Agent executes masks with a frozen vision backbone, and an Evaluator performs iterative refinements, optionally augmented by retrieval of external knowledge. The authors introduce ReasonSeg-Hard to stress-test difficult scenarios and demonstrate competitive or state-of-the-art performance on ReasonSeg benchmarks, with ablations confirming benefits from self-correction and CoT depth. The work highlights the promise of inference-time reasoning to enhance segmentation robustness in challenging, knowledge-dependent contexts, while acknowledging the overhead of invoking large LLMs during inference.

Abstract

Existing works of reasoning segmentation often fall short in complex cases, particularly when addressing complicated queries and out-of-domain images. Inspired by the chain-of-thought reasoning, where harder problems require longer thinking steps/time, this paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results, in the same way humans approach harder questions. We introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction. Instead of fine-tuning, CoT-Seg leverages the inherent reasoning ability of pre-trained MLLMs (GPT-4o) to decompose queries into meta-instructions, extract fine-grained semantics from images, and identify target objects even under implicit or complex prompts. Moreover, CoT-Seg incorporates a self-correction stage: the model evaluates its own segmentation against the original query and reasoning trace, identifies mismatches, and iteratively refines the mask. This tight integration of reasoning and correction significantly improves reliability and robustness, especially in ambiguous or error-prone cases. Furthermore, our CoT-Seg framework allows easy incorporation of retrieval-augmented reasoning, enabling the system to access external knowledge when the input lacks sufficient information. To showcase CoT-Seg's ability to handle very challenging cases ,we introduce a new dataset ReasonSeg-Hard. Our results highlight that combining chain-of-thought reasoning, self-correction, offers a powerful paradigm for vision-language integration driven segmentation.

Figures (17)

• Figure 1: Finding the first violinist (concertmaster) is challenging among similar-looking musicians. CoT-Seg reasons that they sit to the conductor's left and generates a meta-query with relevant spatial information, enabling more accurate segmentation than LISA and Vision Reasoner (No self-correction was needed).
• Figure 2: CoT-Seg reasons about the user’s query to segment surfers in the correct pose, capturing only those who have popped up and are riding waves, unlike LISA and Vision Reasoner (No self-correction was needed).
• Figure 3: CoT-Seg identifies the gym equipment matching the user's query for biceps, e.g., the preacher’s curl, reasoning about its function without any training (Self-correction was needed).
• Figure 4: CoT-Seg reasons about the arrangement of dumbbells to segment those that are unracked, a more challenging task than simple detection (No self-correction was needed).
• Figure 5: Overview of CoT-Seg. The pre-trained MLLM Reasoner generates a chain-of-thought (CoT) over the input image and query, producing an explicit meta-query that translates complex, implicit instructions into clear segmentation guidance. The Segmentation Agent predicts the initial mask, which is then optionally refined by the iterative refinement pipeline. The first-turn mask and original image are examined by the MLLM Evaluator which evaluates the mask and decide if any refinement is necessary. If it does require refinement, then it is passed onto the MLLM Refiner which produces two queries to correct for false positives and negatives. These queries are used inline with the segmentation agent to produce a refined mask for the next iteration of refinement.