Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models

TL;DR

This work tackles the efficiency bottlenecks of Vision-Language Models when processing video inputs by introducing Focus, a streaming concentration architecture. Focus uses a multilevel concentration strategy—semantic token pruning guided by prompts, block-level spatiotemporal concentration, and vector-level redundancy elimination—co-designed with GEMM tiling to execute entirely on-chip. The hardware unit, comprising a Semantic Concentrator and a Similarity Concentrator, enables streaming, tile-local compression with minimal area and negligible runtime impact. Empirically, Focus achieves up to 2.4x speedup and 3.3x energy reduction over baselines, with average sparsity around 80%, and maintains accuracy within about 1.2% of uncompressed models. The approach delivers substantial practical impact for edge-to-cloud deployment of VLMs and is released as open-source so researchers can reproduce and extend the results.

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on tasks such as video captioning and visual question answering. However, their growing scale and video-level inputs lead to significant computational and memory overhead, posing challenges for real-time deployment on hardware accelerators. While prior work attempts to reduce redundancy via token pruning or merging, these methods typically operate at coarse granularity and incur high runtime overhead due to global token-level operations. In this study, we propose Focus, a Streaming Concentration Architecture that efficiently accelerates VLM inference through progressive, fine-grained redundancy elimination. Focus introduces a multilevel concentration paradigm that hierarchically compresses vision-language inputs at three levels: (1) semantic-guided token pruning based on textual prompts, (2) spatial-temporal block-level concentration using localized comparisons, and (3) vector-level redundancy removal via motion-aware matching. All concentration steps are tightly co-designed with the architecture to support streaming-friendly, on-chip execution. Focus leverages GEMM tiling, convolution-style layout, and cross-modal attention to minimize off-chip access while enabling high throughput. Implemented as a modular unit within a systolic-array accelerator, Focus achieves a 2.4x speedup and 3.3x reduction in energy, significantly outperforming state-of-the-art accelerators in both performance and energy efficiency. Full-stack implementation of Focus is open-sourced at https://github.com/dubcyfor3/Focus.

Figures (13)

• Figure 1: Overview of the streaming multilevel concentration architecture.
• Figure 2: Motivation for multilevel concentration. (a) Prompt-aware attention heatmaps. (b) Cosine similarity CDFs. (c) Sparsity Comparison.
• Figure 3: (a) Global token-wise methods (e.g., CMC) perform compression off-chip after writing all token outputs to DRAM. (b) Focus compresses locally and on-chip at the vector level, immediately after each tile is produced.
• Figure 4: Overview of the Focus architecture. The Focus Unit integrates a Semantic Concentrator (SEC) and a Similarity Concentrator (SIC), positioned between compute stages to eliminate redundancy before memory write-back. Both modules operate in a streaming manner and run entirely on-chip.
• Figure 5: Overview of the Semantic Concentrator (SEC), including the streaming importance analyzer, top-$k$ sorter, and offset encoder.