Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLLaVA: An RL-central Framework for Language and Vision Assistants

Lei Zhao, Zihao Ma, Boyu Lin, Yuhe Liu, Wenjun Wu, Lei Huang

TL;DR

RLLaVA addresses the need for an accessible, RL-centered framework for multi-modal language-vision models that can operate on modest hardware. By formulating joint visual-text tasks as an MDP and decoupling algorithmic logic from model architecture and execution engines, it enables rapid experimentation with diverse RL methods while maintaining resource efficiency. Empirical results demonstrate consistent improvements over base models across math, counting, grounding, and agentic tasks, with competitive performance relative to engineering-heavy frameworks. This framework lowers the barrier for research groups with limited GPU resources and offers a flexible path toward broader multimodal RL research and applications.

Abstract

We present an RL-central framework for Language and Vision Assistants (RLLaVA) with its formulation of Markov decision process (MDP). RLLaVA decouples RL algorithmic logic from model architecture and distributed execution, supporting researchers in implementing new RL algorithms with minimal code, and to plug in a broad family of RL methods and vision-language models (VLMs) while remaining agnostic to specific training and inference engines. RLLaVA makes resource-efficient training of 1B--7B models feasible on common GPUs; notably, 4B-scale models can be trained end-to-end with full-parameter updates on a single 24GB GPU. Experiments on multi-modal and agentic tasks demonstrate that RLLaVA has task extensibility, and the models trained with it consistently improve performance over base models, competitive with other specially engineered RL frameworks. The code is available at https://github.com/TinyLoopX/RLLaVA.

RLLaVA: An RL-central Framework for Language and Vision Assistants

TL;DR

RLLaVA addresses the need for an accessible, RL-centered framework for multi-modal language-vision models that can operate on modest hardware. By formulating joint visual-text tasks as an MDP and decoupling algorithmic logic from model architecture and execution engines, it enables rapid experimentation with diverse RL methods while maintaining resource efficiency. Empirical results demonstrate consistent improvements over base models across math, counting, grounding, and agentic tasks, with competitive performance relative to engineering-heavy frameworks. This framework lowers the barrier for research groups with limited GPU resources and offers a flexible path toward broader multimodal RL research and applications.

Abstract

We present an RL-central framework for Language and Vision Assistants (RLLaVA) with its formulation of Markov decision process (MDP). RLLaVA decouples RL algorithmic logic from model architecture and distributed execution, supporting researchers in implementing new RL algorithms with minimal code, and to plug in a broad family of RL methods and vision-language models (VLMs) while remaining agnostic to specific training and inference engines. RLLaVA makes resource-efficient training of 1B--7B models feasible on common GPUs; notably, 4B-scale models can be trained end-to-end with full-parameter updates on a single 24GB GPU. Experiments on multi-modal and agentic tasks demonstrate that RLLaVA has task extensibility, and the models trained with it consistently improve performance over base models, competitive with other specially engineered RL frameworks. The code is available at https://github.com/TinyLoopX/RLLaVA.
Paper Structure (30 sections, 6 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 30 sections, 6 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary and Formulation
  3. Reinforcement Learning for Sequential Decision
  4. Sequential Decision.
  5. Reinforcement Learning.
  6. Policy with Parameterized function.
  7. LLM as Sequential Decision.
  8. Optimization of LLM.
  9. Adaptation for Off-Policy.
  10. Regularization.
  11. RL-Central Framework for Language and Vision Assistants
  12. Formulation of RLLaVA
  13. Architecture Design of RLLaVA
  14. Decoupled Modules
  15. Training Pipeline
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Overview of RLLaVA architecture, showing the RL-centric design with three decoupled dimensions: algorithmic logic (with customizable pipeline), model architecture, and distributed execution strategies.
  • Figure 2: Performance comparison on agentic-search (left) and agentic-coding (right) tasks. The models trained through our framework achieve competitive or superior performance compared to the models from Visual-ARFT liu2025visualarft.