BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization

TL;DR

BUS addresses the inefficiency of ViT-based vision-language pre-training by introducing a bottom-up patch summarization pipeline that first selects text-relevant patches within the ViT backbone (TSPS) and then refines a compact visual summary with a lightweight Patch Abstraction Decoder (PAD). This approach reduces visual token length while preserving cross-modal alignment, enabling higher image resolutions to boost performance without increasing compute. A novel Patch Text Matching (PTM) pre-training objective guides the TSPS to align patches with textual descriptions, complementing standard ITC/ITM/MLM/PrefixLM losses. Experiments across VQA, image captioning, image-text retrieval, and visual grounding demonstrate substantial efficiency gains (e.g., ~51% faster inference) and competitive or state-of-the-art results at higher resolutions, highlighting practical impact for scalable vision-language learning.

Abstract

Vision Transformer (ViT) based Vision-Language Pre-training (VLP) models have demonstrated impressive performance in various tasks. However, the lengthy visual token sequences fed into ViT can lead to training inefficiency and ineffectiveness. Existing efforts address the challenge by either bottom-level patch extraction in the ViT backbone or top-level patch abstraction outside, not balancing training efficiency and effectiveness well. Inspired by text summarization in natural language processing, we propose a Bottom-Up Patch Summarization approach named BUS, coordinating bottom-level extraction and top-level abstraction to learn a concise summary of lengthy visual token sequences efficiently. Specifically, We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction and then attach a flexible Transformer-based Patch Abstraction Decoder (PAD) upon the backbone for top-level visual abstraction. This bottom-up collaboration enables our BUS to yield high training efficiency while maintaining or even improving effectiveness. We evaluate our approach on various visual-language understanding and generation tasks and show competitive downstream task performance while boosting the training efficiency by 50\%. Additionally, our model achieves state-of-the-art performance on many downstream tasks by increasing input image resolution without increasing computational costs over baselines.

Figures (6)

• Figure 1: Subfigure (a) shows the examples of selected text-relevant patch in the VQA scenario. Subfigure (b) shows the overview of our proposed bottom-up patch summarization.
• Figure 2: (a) Overview of our VLP model BUS . (b) The details of the Bottom-Up Patch Summarization.
• Figure 3: The visualization of the VQA test-dev result and Throughput of different VLP models.
• Figure 4: VQA performance and throughput of BUS on different selection ratio of Text-Guided Patch Abstraction.
• Figure 5: The visualization of the VQA case and the selected text-relevant image patches.