StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond

TL;DR

StrucTexTv3 introduces an efficient vision-language model for text-rich image understanding by integrating a Swin-Large visual encoder, a multi-granularity token sampler, and a compact 1.8B LLM. Trained on TIM-30M, a large-scale data collection combining perception and comprehension data with instruction-style prompts, the model delivers state-of-the-art perception results (text spotting, document and chart parsing) and strong comprehension performance (KIE, doc-VQA, table understanding, text translation) while maintaining edge-device deployability. The approach emphasizes high-resolution input handling, multi-scale feature fusion, and unified, instruction-driven learning to tackle diverse tasks. The results demonstrate the practicality and robustness of an efficient VL model for broad real-world text-rich image understanding and its potential for deployment in resource-constrained environments.

Abstract

Text-rich images have significant and extensive value, deeply integrated into various aspects of human life. Notably, both visual cues and linguistic symbols in text-rich images play crucial roles in information transmission but are accompanied by diverse challenges. Therefore, the efficient and effective understanding of text-rich images is a crucial litmus test for the capability of Vision-Language Models. We have crafted an efficient vision-language model, StrucTexTv3, tailored to tackle various intelligent tasks for text-rich images. The significant design of StrucTexTv3 is presented in the following aspects: Firstly, we adopt a combination of an effective multi-scale reduced visual transformer and a multi-granularity token sampler (MG-Sampler) as a visual token generator, successfully solving the challenges of high-resolution input and complex representation learning for text-rich images. Secondly, we enhance the perception and comprehension abilities of StrucTexTv3 through instruction learning, seamlessly integrating various text-oriented tasks into a unified framework. Thirdly, we have curated a comprehensive collection of high-quality text-rich images, abbreviated as TIM-30M, encompassing diverse scenarios like incidental scenes, office documents, web pages, and screenshots, thereby improving the robustness of our model. Our method achieved SOTA results in text-rich image perception tasks, and significantly improved performance in comprehension tasks. Among multimodal models with LLM decoder of approximately 1.8B parameters, it stands out as a leader, which also makes the deployment of edge devices feasible. In summary, the StrucTexTv3 model, featuring efficient structural design, outstanding performance, and broad adaptability, offers robust support for diverse intelligent application tasks involving text-rich images, thus exhibiting immense potential for widespread application.

Figures (2)

• Figure 1: General text-rich image perception and comprehension capabilities of StrucTexTv3. The first row displays perceptual-level capabilities such as text spotting, document parsing, and chart parsing. The second row presents cognitive-level abilities, including document-oriented VQA, key information extraction, table summarization, and text image translation.
• Figure 2: The overview of StrucTexTv3. It comprises three core parts: Visual Encoder, MG-Sampler, and LLM.