TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning

TL;DR

The paper tackles the challenge of instilling accurate visual grounding in pretrained MLLMs without forgetting their existing image-language understanding. It introduces TWIST, a twin-expert, stepwise fine-tuning framework that adds a trainable grounding MoE alongside a frozen image-understanding MoE, with a gating mechanism and stepwise loss to preserve prior capabilities. SCOUT, a synthetic grounded chain-of-thought dataset, provides rich, spatially-aware supervision to strengthen grounding and reasoning. An evaluation protocol across object localization, grounded image captioning, and visual grounding demonstrates that TWIST & SCOUT achieve superior grounding performance while maintaining or improving image understanding, offering a scalable approach to augment MLLMs with spatial reasoning without full retraining.

Abstract

Spatial awareness is key to enable embodied multimodal AI systems. Yet, without vast amounts of spatial supervision, current Multimodal Large Language Models (MLLMs) struggle at this task. In this paper, we introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability without forgetting their existing image and language understanding skills. To this end, we propose TWIST, a twin-expert stepwise tuning module that modifies the decoder of the language model using one frozen module pre-trained on image understanding tasks and another learnable one for visual grounding tasks. This allows the MLLM to retain previously learned knowledge and skills, while acquiring what is missing. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT, which mimics human reasoning in visual grounding. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process, thereby simplifying the task of visual grounding. We evaluate our approach on several standard benchmark datasets, encompassing grounded image captioning, zero-shot localization, and visual grounding tasks. Our method consistently delivers strong performance across all tasks, while retaining the pre-trained image understanding capabilities.

Figures (10)

• Figure 1: TWIST & SCOUT contributions. Our contributions include (a) TWIST, a framework that fine-tunes a pre-trained caption-based MLLM to acquire new grounding skills while retaining existing image understanding capabilities, (b) SCOUT, a scalable synthetic dataset that enhances model performance through step-by-step grounded chain-of-thought annotations, and (c) an evaluation protocol tailored for assessing MLLMs on grounded image captioning tasks.
• Figure 2: TWIST system overview.(a) The MLLM processes an image and text prompt via a vision encoder and language decoder to generate outputs, (b) Twin-Expert, featuring two parallel mixture of experts modules: a frozen one for image understanding and a trainable one for visual grounding, (c) The stepwise loss function breaks down complex reasoning into sequential subtasks, simplifying the training process, (d) During inference, information flows through the image understanding module (blue box) for those tasks and through both modules (blue and red box) for grounding tasks.
• Figure 3: SCOUT data generation. We use an LLM to generate "what" and "where" questions from the input caption. With these questions and the image, we prompt an MLLM to produce the SCOUT grounding dataset, ensuring visually grounded and contextually relevant data.
• Figure A.1: Visualization of the Grounded Chain-of-Thought Dataset. Here we provide step-by-step answers to questions, simplifying the learning process. For instance, identifying the man, finding his hat, determining its color, and finally answering that the hat is orange. This structured approach aids in faster and more effective learning for smaller models.
• Figure A.2: Visualization of Negative Samples. Here we include examples where the question is incorrect or irrelevant, such as asking "What is the cat doing near the shoreline?" when no cat is present. The model begins by identifying the main object and, if it cannot find the object, declares the question invalid. This negative supervision enhances the model's ability to handle invalid or contradictory queries, improving robustness and accuracy.