Unified Reinforcement and Imitation Learning for Vision-Language Models

TL;DR

RIL addresses the need for high-performing yet resource-efficient vision-language systems by unifying reinforcement learning with imitation learning. It leverages a discriminator to measure similarity between student and multiple large teacher outputs and combines this with an accuracy reward from an LLM-as-a-Judge, optimizing a GRPO-based objective augmented by a KL penalty: $\max_{\theta} \frac{1}{2G} \sum_{i=1}^{2G} \{ \min[ r_i(\theta) \hat{A}_i, \mathrm{clip}( r_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i ] - \beta \mathrm{KL}(\pi_{\theta} \| \pi_{ref}) \}$. By using multiple teacher VLMs and including both similarity and answer rewards, RIL enables small VLMs to closely mimic, and in some cases exceed, the performance of large closed/open-source VLMs while preserving inference speed. Extensive experiments across diverse vision-language benchmarks show that RIL narrows the gap to state-of-the-art models and delivers robust results across tasks from recognition to document understanding. The framework is broadly compatible with different image encoders and tokenizers, offering practical benefits for deploying lightweight, high-quality VLMs on edge devices.

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is an LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.

Figures (7)

• Figure 1: Showing the performance improvements (%) of Qwen2.5-VL-7B bai2025qwen2 across vision-language evaluation benchmarks for AI2D kembhavi2016diagram, MathVista lu2023mathvista, MM-Vet yu2023mm, MMMU yue2023mmmu, BLINK fu2024blink, and the average scores for 14 evaluation benchmarks used in Table \ref{['tab:1']}. Note that, we conduct RL on GRPO guo2025deepseek and advanced GRPO, Dr.GRPO liu2025understanding, with only answer rewards from LLM-as-a-Judge zheng2023judging (see Algorithm \ref{['alg:1']}), and we present RIL based on similarity rewards from single or multi large teacher VLMs and simultaneously answer rewards (see Algorithm \ref{['alg:2']}).
• Figure 2: Comparing RIL-applied VLMs based on multi large VLMs with diverse open- and closed-source VLMs, under average performance of numerous vision-language evaluation benchmarks: AI2D kembhavi2016diagram, ChartQA masry2022chartqa, MathVista lu2023mathvista, MMB liu2023mmbench, MM-Vet yu2023mm, MMMU yue2023mmmu, MMMU-Pro yue2024mmmu, MMStar chen2024we, BLINK fu2024blink, SEED li2023seed, SEED2+ li2024seed, and RealWorldQA (RWQA).
• Figure 3: Input prompt for Discriminator
• Figure 4: Input prompt for LLM-as-a-Judge zheng2023judging
• Figure 5: Illustrating training dynamics of small VLMs during RIL, where (Left) showing the evolution of similarity rewards over training iterations and (Mid) accuracy rewards obtained using LLM-as-a-Judge zheng2023judging, ensuring that generated responses are both contextually appropriate and factually correct. (Right) displaying the overall average performance of evaluation benchmarks used in Table \ref{['tab:1']}.