SegLLM: Multi-round Reasoning Segmentation

TL;DR

SegLLM tackles the challenge of multi-round interactive segmentation by introducing a mask-aware, memory-enabled large language model that reintegrates past segmentation outputs into the input stream and propagates conversation context to the mask decoder. The approach relies on two novel components: a Mask-Encoding scheme that embeds reference masks back into the LLM input, and a Mask-Aware Decoding scheme that conditions the mask decoder on both textual history and prior masks. The authors curate MRSeg, a comprehensive multi-round interactive segmentation dataset spanning hierarchical, positional, and interactional relationships, and demonstrate that SegLLM outperforms prior state-of-the-art methods by roughly 18–30% on MRSeg across rounds, while also boosting single-round RES/REC performance by several percentage points. Across benchmarks, SegLLM shows increased robustness to question templates and enhances reasoning about object relationships over multiple rounds, indicating strong potential for practical chat-like segmentation tasks and broader instruction-following segmentation applications.

Abstract

We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation to previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLLM to respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi-round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-round reasoning segmentation data enhances performance on standard single-round referring segmentation and localization tasks, resulting in a 5.5% increase in cIoU for referring expression segmentation and a 4.5% improvement in Acc@0.5 for referring expression localization.

Figures (9)

• Figure 1: We present SegLLM, a multi-round interactive reasoning segmentation model designed to engage in chat-like interactions by responding to both visual and text queries. It reasons about previously segmented objects and conversations to understand complex user intentions. On the left: SegLLM can infer intricate relationships between objects, such as positional, interactional, and hierarchical connections with previously identified entities, e.g., instance [1]. On the right: We introduce the MRSeg, a new multi-round image referring segmentation benchmark. As the rounds progress, the complexity of interaction and memory retention increases, leading to a decline in performance as measured by cIoU. However, SegLLM consistently surpasses the previous state-of-the-art method LISA lai2024lisa, with a significant margin across all conversational rounds.
• Figure 2: Pipeline for generating our multi-round conversational dataset MRSeg. The workflow involves selecting instances, generating relationships, fitting the instances and relationships into conversational templates, and refining the conversations using a language model for improved accuracy.
• Figure 3: Statistics and sample conversations for the Multi-Round Referring Segmentation dataset (MRSeg). We provide more details for MRSeg in \ref{['sec:appendix-dataset-details']}.
• Figure 4: Model architecture of SegLLM for multi-round interactive image reasoning segmentation, which can understand complex user intentions and segment entities based on their relationships with previously identified ones. To facilitate this, first, we implement a mask encoding scheme that reincorporates the reference mask information back into the input stream of the LLMs. This enables the LLMs to reason about segmented masks from previous rounds. Second, we develop a mask-aware decoding scheme that allows the mask decoder to generate new masks based on both the output from the LLMs and the historical memory of output masks. The model uses the last layer hidden states associated with the [REF] and [SEG] tokens to generate both the reference mask and the target mask, seamlessly integrating past and current segmentation results.
• Figure 5: Side-by-side qualitative comparison with LISA's lai2024lisa on multi-round interactive segmentation. SegLLM not only excels in reasoning segmentation, demonstrating an understanding of world knowledge including recognition of famous individuals, as illustrated in the round 1 and round 4 results of the first demo in row one, but it also efficiently responds to questions that reference previous rounds. Ref indicates the referenced output from previous round.