Vision-Language Intelligence: Tasks, Representation Learning, and Large Models

TL;DR

This survey tracks vision-language intelligence from task-specific methods to vision-language pre-training and finally to large-scale, weakly labeled data models. It identifies core architectural choices (dual-stream vs single-stream), embedding schemes (region/grid/patch), and training objectives (ITM, MLM, MVM) that drive cross-modal understanding. It highlights the shift toward scalable, zero-shot capable models (e.g., CLIP, ALIGN, DALL-E) and discusses future directions in modality cooperation, unified representations, and knowledge integration. The work serves as a comprehensive roadmap for researchers and practitioners aiming to develop robust VL systems that generalize across tasks and modalities.

Abstract

This paper presents a comprehensive survey of vision-language (VL) intelligence from the perspective of time. This survey is inspired by the remarkable progress in both computer vision and natural language processing, and recent trends shifting from single modality processing to multiple modality comprehension. We summarize the development in this field into three time periods, namely task-specific methods, vision-language pre-training (VLP) methods, and larger models empowered by large-scale weakly-labeled data. We first take some common VL tasks as examples to introduce the development of task-specific methods. Then we focus on VLP methods and comprehensively review key components of the model structures and training methods. After that, we show how recent work utilizes large-scale raw image-text data to learn language-aligned visual representations that generalize better on zero or few shot learning tasks. Finally, we discuss some potential future trends towards modality cooperation, unified representation, and knowledge incorporation. We believe that this review will be of help for researchers and practitioners of AI and ML, especially those interested in computer vision and natural language processing.

Figures (7)

• Figure 1: The three stages of task specific methods. The main differences are the granularity of the visual representation and the way of fusing vision and language features.
• Figure 2: The img2seq structure contains an image encoder such as a CNN and a language decoder such as an LSTM.
• Figure 3: The architecture of vanilla VQA antol2015vqa contains a CNN model to encode the input images and an LSTM model to encode the input question. The encoded image and question features are merged with dot product and then go through a fully connected layer to predict the probability over candidate answers.
• Figure 4: The overview of the Deep fragmentkarpathy2014deep architecture. Left: Detected objects are mapped to fragment embedding space. Right: Dependence tree relations are encoded to fragment embedding space.
• Figure 5: (a) Original BERT with single-modality, where some language tokens are masked for prediction to train language representation. (b) Modified BERT with multi-modality, where both image and language tokens are fed into a BERT-like Transformer model.