Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review

Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Jiayao Ma, Qianyi Jiang, Kai Zhou, Junfeng Luo

TL;DR

This survey addresses the TIU problem by assessing Multimodal Large Language Models through three core architecture components (Visual Encoder, Modality Connector, LLM Decoder) and a three-stage training pipeline (Modality Alignment, Instruction Alignment, Preference Alignment). It systematically categorizes encoders (OCR-free, OCR-based, mixture), modality fusion strategies, and LLM choices, while mapping datasets and benchmarks across domain-specific and comprehensive TIU tasks. The paper reports that OCR-free encoders currently lead SOTA performance, discusses efficiency bottlenecks and token-length challenges, and highlights long-document and multilingual TIU as pressing frontiers. Overall, it provides a structured, multi-dimensional view of progress, limitations, and future directions to guide researchers and practitioners in designing more capable and scalable TIU MLLMs.

Abstract

The recent emergence of Multi-modal Large Language Models (MLLMs) has introduced a new dimension to the Text-rich Image Understanding (TIU) field, with models demonstrating impressive and inspiring performance. However, their rapid evolution and widespread adoption have made it increasingly challenging to keep up with the latest advancements. To address this, we present a systematic and comprehensive survey to facilitate further research on TIU MLLMs. Initially, we outline the timeline, architecture, and pipeline of nearly all TIU MLLMs. Then, we review the performance of selected models on mainstream benchmarks. Finally, we explore promising directions, challenges, and limitations within the field.

Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review

TL;DR

This survey addresses the TIU problem by assessing Multimodal Large Language Models through three core architecture components (Visual Encoder, Modality Connector, LLM Decoder) and a three-stage training pipeline (Modality Alignment, Instruction Alignment, Preference Alignment). It systematically categorizes encoders (OCR-free, OCR-based, mixture), modality fusion strategies, and LLM choices, while mapping datasets and benchmarks across domain-specific and comprehensive TIU tasks. The paper reports that OCR-free encoders currently lead SOTA performance, discusses efficiency bottlenecks and token-length challenges, and highlights long-document and multilingual TIU as pressing frontiers. Overall, it provides a structured, multi-dimensional view of progress, limitations, and future directions to guide researchers and practitioners in designing more capable and scalable TIU MLLMs.

Abstract

The recent emergence of Multi-modal Large Language Models (MLLMs) has introduced a new dimension to the Text-rich Image Understanding (TIU) field, with models demonstrating impressive and inspiring performance. However, their rapid evolution and widespread adoption have made it increasingly challenging to keep up with the latest advancements. To address this, we present a systematic and comprehensive survey to facilitate further research on TIU MLLMs. Initially, we outline the timeline, architecture, and pipeline of nearly all TIU MLLMs. Then, we review the performance of selected models on mainstream benchmarks. Finally, we explore promising directions, challenges, and limitations within the field.

Paper Structure

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Visual Encoder
  4. Modality Connector
  5. LLM Decoder
  6. Training Pipeline
  7. Modality Alignment
  8. Instruction Alignment
  9. Preference Alignment
  10. Datasets and Benchmarks
  11. Challenges and Trends
  12. Limitations

Figures (3)

  • Figure 1: The development timeline of TIU MLLMs.
  • Figure 2: The general model architecture of MLLMs and the implementation choices for each component.
  • Figure 3: The evolutionary tree of modern LLMs traces the development of language models in recent years and highlights some of the most well-known models. According to the classification of Encoders, the blue branch is ocr-free, the pink branch is ocr-based, and the green branch is Mixture of Encoders.