Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Sihan Cao, Jianwei Zhang, Pengcheng Zheng, Jiaxin Yan, Caiyan Qin, Yalan Ye, Wei Dong, Peng Wang, Yang Yang, Chaoning Zhang

Abstract

Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7\% of visual tokens and achieves up to a 54.2\% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7\%. Code is released at \href{https://github.com/MagicVicCoder/TPRL}{\textcolor{mypink}{https://github.com/MagicVicCoder/TPRL}}.

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Abstract

Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7\% of visual tokens and achieves up to a 54.2\% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7\%. Code is released at \href{https://github.com/MagicVicCoder/TPRL}{\textcolor{mypink}{https://github.com/MagicVicCoder/TPRL}}.
Paper Structure (18 sections, 23 equations, 4 figures, 6 tables)

This paper contains 18 sections, 23 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models
  4. Visual Token Pruning
  5. Reinforcement Learning for Model Compression
  6. Methodology
  7. Overview
  8. Language-guided MDP Formulation
  9. Learning from Demonstrations
  10. RL-based Adaptive Token Pruning
  11. Training and Inference
  12. Experiments
  13. Experimental Setup
  14. Implementation Details
  15. Datasets and Metrics
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Overview of the TPRL framework consisting of three synergistic stages. (a) Autoencoder training for state compression learns a compact latent representation to reduce the feature dimensionality of each visual token. (b) Learning from demonstrations provides a foundational initialization for the pruning policy by imitating high-quality trajectories to ensure stable convergence. (c) RL training using PPO fine-tunes the policy through language-guided sequential optimization to achieve an optimal balance between task performance and computational efficiency.
  • Figure 2: Impact of state compression dimension $d_l$ on relative accuracy across RL training iterations.
  • Figure 3: Comparison of RL optimization methods.
  • Figure 4: Qualitative visualization of sequential pruning trajectories via mask superimposition. The agent progressively filters backgrounds and redundant details to isolate task-critical information through our multi-step MDP formulation.