From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information

TL;DR

This work tackles the challenge of enabling Multimodal Large Language Models (MLLMs) to better understand fine-grained visual details by infusing textual detection outputs from OD/OCR models. It systematically compares training-free infusion (TFI), retraining-based infusion (RBI), and fine-tuning-based infusion (FTBI) across multiple MLLMs and detectors, evaluating on ten diverse benchmarks. The results show that FTBI yields the strongest and most consistent gains (up to 6.71% relative improvement over TFI) and remains effective when the detection model is replaced with open-set detectors, indicating robust transferability. The study provides practical guidelines for adaptive training in fusion strategies and releases code to foster further exploration of detection-informed multimodal reasoning.

Abstract

Despite the impressive capabilities of Multimodal Large Language Models (MLLMs) in integrating text and image modalities, challenges remain in accurately interpreting detailed visual elements. Vision detection models excel at recognizing fine-grained image details, prompting researchers to use them to enhance MLLMs. One effective strategy is to infuse detection information in text format, which has proven simple and effective. However, most studies utilize this method without training, leaving the potential of adaptive training largely unexplored. Adaptive training could significantly enhance MLLMs' comprehension of unique inputs while filtering out irrelevant information. This paper addresses the crucial question: How does training impact MLLMs' understanding of infused textual detection information? We systematically experiment with various representative models to evaluate the effects of training-free, retraining, and fine-tuning strategies. We also examine the influence of training on MLLMs' original abilities and the interchangeability of detection models. Our findings indicate that fine-tuning a pre-trained MLLM to incorporate textual detection information delivers superior results compared to training-free and retraining methods, improving performance by 6.71% across 10 widely recognized benchmarks. Furthermore, fine-tuning enables MLLMs to retain performance enhancements even when detection models are swapped, indicating improved understanding of formatted textual data. We release our codes to support further exploration of fusion strategies for vision detection models and the enhancement of MLLMs' fine-grained multimodal capabilities.

Figures (7)

• Figure 1: Examples where LLaVA-1.5-13B fails, while the model infused with textual detection information (FTBI-13B) succeeds. "Detection" refers to processed detection information from OD/OCR models. Additional examples are provided in Figure \ref{['appendix-case-we-work']} of Appendix \ref{['sec-appendix-case-we-work']}.
• Figure 2: The studied MLLM architectures with different training strategies for infusing textual detection information. "(LLaVA-1.5)" denotes module initialization with weights from LLaVA-1.5.
• Figure 3: An example of detecting open-set targets with Grounding DINO.
• Figure 4: Examples on which LLaVA-1.5 fails while the fine-tune model (FTBI-13B) with open-set object detection information succeeds.
• Figure 5: Examples on which LLaVA-1.5-13B fails while the model infused with textual detection information (FTBI-13B) succeeds.