Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

TL;DR

This survey addresses the limitation of text-only LLMs by organizing Vision-Language Models (VLMs) into a practical taxonomy based on input and output capabilities. It analyzes architectures, data sources, and strengths/weaknesses across about 70 VLMs and benchmarks them on standard datasets including the MME suite. The work highlights design tradeoffs, common building blocks (encoders, fusion modules, and LLMs), and outlines future directions in modularity, evaluation, multilingual grounding, and safety. The resulting roadmap aims to guide researchers and practitioners in building robust, efficient, and safe multimodal AI systems across vision, language, and beyond.

Abstract

The advent of Large Language Models (LLMs) has significantly reshaped the trajectory of the AI revolution. Nevertheless, these LLMs exhibit a notable limitation, as they are primarily adept at processing textual information. To address this constraint, researchers have endeavored to integrate visual capabilities with LLMs, resulting in the emergence of Vision-Language Models (VLMs). These advanced models are instrumental in tackling more intricate tasks such as image captioning and visual question answering. In our comprehensive survey paper, we delve into the key advancements within the realm of VLMs. Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.This classification is based on their respective capabilities and functionalities in processing and generating various modalities of data.We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible, providing readers with a comprehensive understanding of its essential components. We also analyzed the performance of VLMs in various benchmark datasets. By doing so, we aim to offer a nuanced understanding of the diverse landscape of VLMs. Additionally, we underscore potential avenues for future research in this dynamic domain, anticipating further breakthroughs and advancements.

Figures (2)

• Figure 1: Taxonomy of Visual Language Models highlighting the input and output formats that the models are capable of handling.
• Figure 2: A high-level overview of the architecture of VLMs highlighting various design choices with corresponding examples.