EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models

TL;DR

This work tackles the challenge of enabling Multimodal LLMs to understand arbitrary referring visual prompts without heavy architectural changes or extensive retraining. It introduces EAGLE, which renders prompts as colored patches on images and leverages Geometry-Agnostic Learning (GAL) to convert diverse region annotations into a uniform set of reference points, disentangling geometry from region semantics. Training uses autoregressive instruction tuning on a fixed base model with LoRA, guided by a reformulated multimodal dataset and an explicit objective $p(oldsymbol{X}_{\mathrm{a}} \mid \boldsymbol{X}_{\mathrm{v}}, \boldsymbol{X}_{\mathrm{instruct}}) = \prod_{i=1}^{L} p_{\boldsymbol{\theta}}(x_i \mid \boldsymbol{X}_{\mathrm{v}}, \boldsymbol{X}_{\mathrm{instruct}}, \boldsymbol{X}_{\mathrm{a},<i})$, enabling efficient generalization to irregular prompts. Empirical results on semantic segmentation benchmarks and arbitrary prompt scenarios show that EAGLE achieves competitive or superior performance with less training, particularly under incomplete or nonstandard prompts. The approach offers practical benefits for real-world multimodal systems by simplifying prompt handling, reducing data requirements, and enhancing robustness to prompt geometry, with broader implications for region-aware visual reasoning in LLM-driven frameworks.

Abstract

Recently, Multimodal Large Language Models (MLLMs) have sparked great research interests owing to their exceptional content-reasoning and instruction-following capabilities. To effectively instruct an MLLM, in addition to conventional language expressions, the practice of referring to objects by painting with brushes on images has emerged as a prevalent tool (referred to as "referring visual prompts") due to its efficacy in aligning the user's intention with specific image regions. To accommodate the most common referring visual prompts, namely points, boxes, and masks, existing approaches initially utilize specialized feature encoding modules to capture the semantics of the highlighted areas indicated by these prompts. Subsequently, these encoded region features are adapted to MLLMs through fine-tuning on a meticulously curated multimodal instruction dataset. However, such designs suffer from redundancy in architecture. Moreover, they face challenges in effectively generalizing when encountering a diverse range of arbitrary referring visual prompts in real-life scenarios. To address the above issues, we propose EAGLE, a novel MLLM that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches. Specifically, our EAGLE maintains the innate format of the referring visual prompts as colored patches rendered on the given image for conducting the instruction tuning. Our approach embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM, with the semantic comprehension of these regions originating from the MLLM itself. Besides, we also propose a Geometry-Agnostic Learning paradigm (GAL) to further disentangle the MLLM's region-level comprehension with the specific formats of referring visual prompts. Extensive experiments are conducted to prove the effectiveness of our proposed method.

Figures (6)

• Figure 1: Performances comparison of prevalent MLLMs and our proposed EAGLE on (a) different datasets and (b) arbitrary referring visual prompts. We leverage semantic segmentation as the proxy task to evaluate the referring visual prompt recognition capabilities of MLLMs following Osprey yuan2023osprey. In (a), we follow the configuration of Osprey by simply using ground-truth masks as the visual prompt. In (b), we imitate arbitrary referring visual prompts by degenerating the integrity of ground-truth masks, in which we demonstrate the consistent performance of our GAL paradigm in dealing with arbitrary referring visual prompts. The area of circles indicates the scale of employed training data.
• Figure 2: Comparison of our proposed Eagle with previous methods. Previous approaches (left column) specifically extract region features and project them into LLM's input space. Our EAGLE (right column) preprocesses arbitrary referring visual prompts with the introduced Geometry-Agnostic Learning (GAL) paradigm, and then renders them onto the image for highlighting the referred regions.
• Figure 3: The overall framework of our proposed EAGLE. Given arbitrary referring visual prompts, our method first transforms them into a set of uniform points to disentangle the later region-level recognition learning with diverse shapes and formats of referring visual prompts. These transformed points are rendered onto the input image to highlight referred regions while not harming the complete semantics of the original image. Afterward, the rendered image and user instructions are fed into the MLLM to generate the final response.
• Figure 4: Illustration of reformulated conversational data with multimodal instructions. For training data, we adopt the ground-truth annotations as referring visual prompts. For test data, we degrade the original annotations to imitate the arbitrary referring visual prompts encountered in real-life scenarios.
• Figure 5: Illustration of different formats of rendered point markers. We evaluate the sensitivity of the MLLM to three colors (red, green, and blue) and four prompt types (dot, square, box, and circle), for a total of 12 combinations.