MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks

TL;DR

MaMMUT presents a decoder-only vision-language model with a single image encoder and a shared text decoder, unified by a two-pass training scheme that enables simultaneous contrastive and autoregressive captioning learning. The approach achieves state-of-the-art or competitive results across image-text/text-image retrieval, VQA, VideoQA, Video Captioning, and Open-Vocabulary Detection, often at a fraction of the size of larger foundations models. Key innovations include the focal contrastive loss, Cropped Positional Embeddings for localization, and a lightweight TubeViT-based extension for video without extra pretraining. The work demonstrates that a simple, weight-sharing architecture can flexibly address diverse multimodal tasks and translate well to video-language domains with minimal architectural changes, offering a practical, scalable vision-language foundation. Overall, MaMMUT advances open-vocabulary, video-language, and retrieval capabilities while maintaining efficiency and broad task coverage.

Abstract

The development of language models have moved from encoder-decoder to decoder-only designs. In addition, we observe that the two most popular multimodal tasks, the generative and contrastive tasks, are nontrivial to accommodate in one architecture, and further need adaptations for downstream tasks. We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. This is done with a simple model, called MaMMUT. It consists of a single vision encoder and a text decoder, and is able to accommodate contrastive and generative learning by a novel two-pass approach on the text decoder. We demonstrate that joint learning of these diverse objectives is simple, effective, and maximizes the weight-sharing of the model across these tasks. Furthermore, the same architecture enables straightforward extensions to open-vocabulary object detection and video-language tasks. The model tackles a diverse range of tasks, while being modest in capacity. Our model achieves the state of the art on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models. It shows very competitive results on VQA and Video Captioning, especially considering its capacity. Ablations confirm the flexibility and advantages of our approach.

Figures (4)

• Figure 1: The MaMMUT model is a simple vision-encoder and text-decoder architecture, which serves as foundational model for both image-language and video-language tasks. Despite its relatively small size, the model outperforms SOTA on many diverse tasks. Example results on Image-Text/Text-Image retrieval, Visual Question Answering (VQA), Open-vocabulary detection, Video Question Answering (VideoQA) are shown.
• Figure 2: MaMMUT model architecture with an image encoder and text decoder (left), compared to others. Many encoder-decoder architectures (center) cannot handle the contrastive objective, for example pmlr-v139-cho21apali. Approaches to combine contrastive and captioning (right), e.g. Align-Before-Fuse (ALBEF) albef or CoCa yu2022coca develop more complex models and are hard to extend to video inputs or localization tasks. Our architecture is simpler than previous approaches and is able to accommodate more tasks.
• Figure 3: MaMMUT two-pass learning. We compute the image features via the image encoder. Then we compute the contrastive text features and loss by applying a non-causal masking to the language decoder, which makes it effectively a text encoder (top). Finally we compute the generative text features and loss by applying causal masking and cross attention with the image features (bottom). The decoder is visualized twice for clarity, its weights are fully shared.
• Figure 4: MaMMUT video model (left) efficiently and seamlessly extends the image-language model, by adding learnable spatio-temporal features. The model is applied only once, as opposed to other models, processing each individual frame independently (right). Our model uses image-text pre-training only.