Optimizing Vision-Language Interactions Through Decoder-Only Models
Kaito Tanaka, Benjamin Tan, Brian Wong
TL;DR
MUDAIF tackles inefficiencies and misalignment in traditional encoder-based VLMs by adopting a decoder-only architecture that uses Vision-Token Adapter to convert visual input into token-like representations and an adaptive co-attention module for bidirectional fusion. Trained on 45M image-text pairs and fine-tuned with multimodal instructions, MUDAIF achieves state-of-the-art results on diverse vision-language benchmarks, including VQA, image captioning, and multimodal reasoning, while offering improved efficiency and scalability due to the encoder-free design. The paper provides extensive ablations, human evaluations, and analyses demonstrating better cross-modal alignment, robustness, and generalization, validating the practicality of encoder-free vision-language modeling. It also points to future directions in extending MUDAIF to video and more complex multimodal interactions, reinforcing its potential as a new standard in encoder-free VLMs.
Abstract
Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.