Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Large Language Models for Generalized and Specialized Applications

Yifan Li, Zhixin Lai, Wentao Bao, Zhen Tan, Anh Dao, Kewei Sui, Jiayi Shen, Dong Liu, Huan Liu, Yu Kong

TL;DR

This survey analyzes visual large language models (VLLMs) as a unifying framework for integrating vision and language across broad and domain-specific tasks. It organizes applications into three axes—vision-to-text, vision-to-action, and text-to-vision—and details concrete methods, datasets, and training regimes enabling multi-task reasoning, perception, and generation. The authors discuss ethical considerations, efficiency, interpretability, and hallucination, and offer directions for security, privacy, and complex reasoning to guide future work. Overall, the paper provides a comprehensive roadmap for advancing VLLMs toward scalable, trustworthy, and broadly applicable multimodal AI systems.

Abstract

Visual-language models (VLM) have emerged as a powerful tool for learning a unified embedding space for vision and language. Inspired by large language models, which have demonstrated strong reasoning and multi-task capabilities, visual large language models (VLLMs) are gaining increasing attention for building general-purpose VLMs. Despite the significant progress made in VLLMs, the related literature remains limited, particularly from a comprehensive application perspective, encompassing generalized and specialized applications across vision (image, video, depth), action, and language modalities. In this survey, we focus on the diverse applications of VLLMs, examining their using scenarios, identifying ethics consideration and challenges, and discussing future directions for their development. By synthesizing these contents, we aim to provide a comprehensive guide that will pave the way for future innovations and broader applications of VLLMs. The paper list repository is available: https://github.com/JackYFL/awesome-VLLMs.

Visual Large Language Models for Generalized and Specialized Applications

TL;DR

This survey analyzes visual large language models (VLLMs) as a unifying framework for integrating vision and language across broad and domain-specific tasks. It organizes applications into three axes—vision-to-text, vision-to-action, and text-to-vision—and details concrete methods, datasets, and training regimes enabling multi-task reasoning, perception, and generation. The authors discuss ethical considerations, efficiency, interpretability, and hallucination, and offer directions for security, privacy, and complex reasoning to guide future work. Overall, the paper provides a comprehensive roadmap for advancing VLLMs toward scalable, trustworthy, and broadly applicable multimodal AI systems.

Abstract

Visual-language models (VLM) have emerged as a powerful tool for learning a unified embedding space for vision and language. Inspired by large language models, which have demonstrated strong reasoning and multi-task capabilities, visual large language models (VLLMs) are gaining increasing attention for building general-purpose VLMs. Despite the significant progress made in VLLMs, the related literature remains limited, particularly from a comprehensive application perspective, encompassing generalized and specialized applications across vision (image, video, depth), action, and language modalities. In this survey, we focus on the diverse applications of VLLMs, examining their using scenarios, identifying ethics consideration and challenges, and discussing future directions for their development. By synthesizing these contents, we aim to provide a comprehensive guide that will pave the way for future innovations and broader applications of VLLMs. The paper list repository is available: https://github.com/JackYFL/awesome-VLLMs.
Paper Structure (33 sections, 8 figures, 1 table)

This paper contains 33 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Texonomy
  3. Image-to-text
  4. General domain application
  5. Specific Domain Application
  6. Video-to-text
  7. General-purpose Video Understanding
  8. Video Conversation
  9. Egocentric Understanding
  10. Vision-to-action
  11. Autonomous Driving
  12. Embodied AI
  13. Automated Tool Management
  14. Text-to-vision
  15. Text-to-image
  16. ...and 18 more sections

Figures (8)

  • Figure 1: The evolution of VLMs includes three phases: conventional VLMs (before 2018), emerging VLMs (2018-2022), and VLLMs (2022 onward).
  • Figure 2: The architectures of three VLLM application categories: (a) vision-to-text, (b) vision-to-action, (c) text-to-vision.
  • Figure 3: VLLM application texonomy. REC (Referring Expression Comprehension), RES (Referring Expression Segmentation), OCR (Optical Character Recognition).
  • Figure 4: An illustration of image-to-text general domain applications.
  • Figure 5: An illustration of image-to-text specific domain applications.
  • ...and 3 more figures