VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models

TL;DR

VisualRWKV introduces VisualRWKV, the first multimodal visual language model built on the linear RNN RWKV to overcome Transformer self-attention’s $O(L^2)$ complexity by delivering linear-time recurrence with $O(L)$ memory and $O(1)$ per-token inference. It combines data-dependent recurrence, a sandwich prompting strategy, and a 2D image scanning mechanism to effectively model visual sequences within a RWKV backbone. Across eight benchmarks, VisualRWKV achieves competitive performance relative to Transformer-based VLMs of similar scale, with substantial efficiency gains (speedup up to 3.98x and ~54% memory savings at long lengths) and notable multilingual capabilities. Ablation studies confirm the value of data-dependent recurrence, the sandwich prompt, and bidirectional/multidirectional image scanning, supporting the broader potential of linear RNNs for multimodal learning. The work provides public checkpoints and code to facilitate further research and rapid evaluation of linear-RNN-based multimodal models.

Abstract

Visual Language Models (VLMs) have rapidly progressed with the recent success of large language models. However, there have been few attempts to incorporate efficient linear Recurrent Neural Networks (RNNs) architectures into VLMs. In this study, we introduce VisualRWKV, the first application of a linear RNN model to multimodal learning tasks, leveraging the pre-trained RWKV language model. We propose a data-dependent recurrence and sandwich prompts to enhance our modeling capabilities, along with a 2D image scanning mechanism to enrich the processing of visual sequences. Extensive experiments demonstrate that VisualRWKV achieves competitive performance compared to Transformer-based models like LLaVA-1.5 on various benchmarks. Compared to LLaVA-1.5, VisualRWKV has a speed advantage of 3.98 times and can save 54% of GPU memory when reaching an inference length of 24K tokens. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at the following GitHub repository: see https://github.com/howard-hou/VisualRWKV.

Figures (6)

• Figure 1: VisualRWKV outperforms the SoTA LLaVA-1.5 liu2023improvedllava on 4 tasks (a), with high computational efficiency (b) and low, stable memory usage (c).
• Figure 2: VisualRWKV architecture overview and three prompting method. Image First Prompt: place image tokens before instruction tokens; Image Last Prompt: place image tokens after instruction tokens; Sandwich Prompt: place image tokens in the middle of instruction tokens. Red words indicate the key contributions.
• Figure 3: Data-dependent recurrence. Top: Semantic diagram of the time-mixing block; Bottom: Time-mixing block as an RNN cell. Dashed arrows represent connections in data-dependent recurrence, not present in data-independent recurrence.
• Figure 4: Illustration of 3 different multimodal RWKV Blocks: Unidirectional Blocks (left), Bidirectional Blocks (middle), and Multidirectional Blocks (right). The four scanning modes are also depicted at the top.
• Figure 5: Single-Stage Training vs. Two-Stage Training. We conducted a comparative analysis between the two-stage training and single-stage training, with the latter omitting the vision-language alignment phase. Our findings reveal that the single-stage training yields inferior performance outcomes. This suggests that the vision-language alignment, integral to the two-stage training, significantly contributes to enhanced performance.