POINTS1.5: Building a Vision-Language Model towards Real World Applications

TL;DR

POINTS1.5 delivers a high-performance vision-language model optimized for real-world tasks by incorporating a NaViT-style dynamic high-resolution vision encoder, expanded Chinese data, and carefully filtered visual instruction tuning datasets. Built on a LLaVA-style architecture with a two-layer MLP projector and the Qwen2.5-7B-Instruct LLM, it employs a three-stage training strategy and model soup to achieve state-of-the-art results among sub-10B models on the OpenCompass leaderboard using fewer than 5B training tokens. The work demonstrates strong capabilities across OCR, chart reasoning, and math, while carefully addressing data quality and bilingual challenges to broaden applicability. It also discusses practical considerations for future scaling with larger LLMs and improved data balancing across categories."

Abstract

Vision-language models have made significant strides recently, demonstrating superior performance across a range of tasks, e.g. optical character recognition and complex diagram analysis. Building on this trend, we introduce a new vision-language model, POINTS1.5, designed to excel in various real-world applications. POINTS1.5 is an enhancement of POINTS1.0 and incorporates several key innovations: i) We replace the original CLIP vision encoder, which had a fixed image resolution, with a NaViT-style vision encoder that supports native dynamic high resolution. This allows POINTS1.5 to process images of any resolution without needing to split them into tiles. ii) We add bilingual support to POINTS1.5, significantly enhancing its capability in Chinese. Due to the scarcity of open-source Chinese datasets for vision-language models, we collect numerous images from the Internet and annotate them using a combination of manual and automatic methods. iii) We propose a set of rigorous filtering methods for visual instruction tuning datasets. We comprehensively evaluate all these filtering methods, and choose the most effective ones to obtain the final visual instruction tuning set. Thanks to these innovations, POINTS1.5 significantly outperforms POINTS1.0 and demonstrates strong performance across a range of real-world applications. Notably, POINTS1.5-7B is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among models with fewer than 10 billion parameters

Figures (21)

• Figure 1: Performance of Open-Source Models on the OpenCompass Leaderboard2023opencompass. POINTS1.5 ranks first among all models under 10B in size, even outperforming models several times larger. The size of each bubble represents the model size.
• Figure 2: POINTS1.5 shows great potential to solve challenging real world problems.
• Figure 3: POINTS1.5 uses the converntional LLaVA-style architecture, consisting of a vision encoder, a MLP projector and a LLM.
• Figure 4: The chat template during pre-training in POINTS1.0 (above) and POINTS1.5 (below)
• Figure 5: Prompts used in the chat template during pre-training stage.