CoReS: Orchestrating the Dance of Reasoning and Segmentation

TL;DR

CoReS tackles the challenge of grounding fine-grained object regions described by reasoning texts in multi-modal images by introducing a dual-chain, top-down framework. It couples a Chain-of-Reasoning with a Chain-of-Segmenting under in-context guidance, guiding the MLLM through a hierarchical output that aligns with segmentation steps and uses a SAM-based mask generator. The method yields a 6.5 percentage-point improvement on the ReasonSeg dataset and demonstrates strong generalization to other benchmarks, with ablations confirming the value of both chain components and the prompting strategy. Overall, the work shows that incorporating a multimodal chain-of-thought can substantially improve dense, fine-grained perception tasks and can generalize across diverse multi-modal contexts.

Abstract

The reasoning segmentation task, which demands a nuanced comprehension of intricate queries to accurately pinpoint object regions, is attracting increasing attention. However, Multi-modal Large Language Models (MLLM) often find it difficult to accurately localize the objects described in complex reasoning contexts. We believe that the act of reasoning segmentation should mirror the cognitive stages of human visual search, where each step is a progressive refinement of thought toward the final object. Thus we introduce the Chains of Reasoning and Segmenting (CoReS) and find this top-down visual hierarchy indeed enhances the visual search process. Specifically, we propose a dual-chain structure that generates multi-modal, chain-like outputs to aid the segmentation process. Furthermore, to steer the MLLM's outputs into this intended hierarchy, we incorporate in-context inputs as guidance. Extensive experiments demonstrate the superior performance of our CoReS, which surpasses the state-of-the-art method by 6.5\% on the ReasonSeg dataset. Project: https://chain-of-reasoning-and-segmentation.github.io/.

Figures (7)

• Figure 1: Comparison between our CoReS and LISA. UP: the process of LISA, DOWN: the diagram of CoReS. Given textual and visual inputs, LISA directly uses the [SEG] token output by MLLM to generate a mask. On the contrary, our CoReS involves breaking down the task of "finding the part that gives dogs keen sense of smell" into a logical chain such as "first find the front part of the dog's face, then focus on this specific area, search for the nose of the dog." It can be observed that LISA incorrectly segments the dog's eyes, which are similarly round, dark, and important in sensory perception. In contrast, through in-context input and dual-chain structure, CoReS achieves the segmentation of the nose of the dog correctly.
• Figure 2: Overall architecture of CoReS. The input of MLLM consists of the user input in gray and the extra in-context input in orange, which consists of question-answer examples unrelated to the user query. MLLM generates output at the logical level of chain-of-reasoning, where the token embeddings of [LOC] and [SEG] serve as prompt inputs for different positions of the segmentation chain, guiding the chain to generate segmentation results progressively. We exclude the visual feature input to the mask decoder extracted by the extra vision backbone for conciseness here.
• Figure 3: Examples of MLLM input. The brown section represents the system prompt commonly used for MLLM. The green section is the in-context input proposed in this paper, and the black section is the user input for questioning the image.
• Figure 4: Visual comparison of CoReS and LISA.
• Figure 5: Qualitative interpretation of the advantages of the multi-modal chain-of-thought over LISA. From left to right are the input image, LISA result, CoReS first logic layer segmentation result, CoReS final result, and ground truth mask.