Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ModRWKV: Transformer Multimodality in Linear Time

Jiale Kang, Ziyin Yue, Qingyu Yin, Jiang Rui, Weile Li, Zening Lu, Zhouran Ji

TL;DR

ModRWKV investigates transforming multimodal large language modeling from transformer-centric to an efficient, linear-time paradigm by building a modular RNN-based framework on RWKV7. It combines lightweight modality encoders (vision, audio, time series), a single-MLP adapter, and 1D sequence compression to fuse diverse signals with the RWKV backbone. Across seven multimodal benchmarks and time-series tasks, ModRWKV achieves competitive or superior results relative to larger vision-language models, while maintaining lower computational demands, aided by pretraining weights (G1) that enhance multimodal understanding. The work demonstrates that modern RNN architectures can be viable alternatives to Transformers for MLLMs, offering practical benefits in efficiency and scalability, and it identifies key design choices that drive performance, such as encoder selection and sequence compression strategies.

Abstract

Currently, most multimodal studies are based on large language models (LLMs) with quadratic-complexity Transformer architectures. While linear models like RNNs enjoy low inference costs, their application has been largely limited to the text-only modality. This work explores the capabilities of modern RNN architectures in multimodal contexts. We propose ModRWKV-a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone-which achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. We designed the multimodal modules in ModRWKV with an extremely lightweight architecture and, through extensive experiments, identified a configuration that achieves an optimal balance between performance and computational efficiency. ModRWKV leverages the pretrained weights of the RWKV7 LLM for initialization, which significantly accelerates multimodal training. Comparative experiments with different pretrained checkpoints further demonstrate that such initialization plays a crucial role in enhancing the model's ability to understand multimodal signals. Supported by extensive experiments, we conclude that modern RNN architectures present a viable alternative to Transformers in the domain of multimodal large language models (MLLMs). Furthermore, we identify the optimal configuration of the ModRWKV architecture through systematic exploration.

ModRWKV: Transformer Multimodality in Linear Time

TL;DR

ModRWKV investigates transforming multimodal large language modeling from transformer-centric to an efficient, linear-time paradigm by building a modular RNN-based framework on RWKV7. It combines lightweight modality encoders (vision, audio, time series), a single-MLP adapter, and 1D sequence compression to fuse diverse signals with the RWKV backbone. Across seven multimodal benchmarks and time-series tasks, ModRWKV achieves competitive or superior results relative to larger vision-language models, while maintaining lower computational demands, aided by pretraining weights (G1) that enhance multimodal understanding. The work demonstrates that modern RNN architectures can be viable alternatives to Transformers for MLLMs, offering practical benefits in efficiency and scalability, and it identifies key design choices that drive performance, such as encoder selection and sequence compression strategies.

Abstract

Currently, most multimodal studies are based on large language models (LLMs) with quadratic-complexity Transformer architectures. While linear models like RNNs enjoy low inference costs, their application has been largely limited to the text-only modality. This work explores the capabilities of modern RNN architectures in multimodal contexts. We propose ModRWKV-a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone-which achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. We designed the multimodal modules in ModRWKV with an extremely lightweight architecture and, through extensive experiments, identified a configuration that achieves an optimal balance between performance and computational efficiency. ModRWKV leverages the pretrained weights of the RWKV7 LLM for initialization, which significantly accelerates multimodal training. Comparative experiments with different pretrained checkpoints further demonstrate that such initialization plays a crucial role in enhancing the model's ability to understand multimodal signals. Supported by extensive experiments, we conclude that modern RNN architectures present a viable alternative to Transformers in the domain of multimodal large language models (MLLMs). Furthermore, we identify the optimal configuration of the ModRWKV architecture through systematic exploration.

Paper Structure

This paper contains 30 sections, 6 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background
  3. RWKV7: Modern RNN Architecture
  4. Multimodal Large Language Models
  5. Methodology
  6. Multimodal Encoder
  7. Vision Encoder.
  8. Audio Encoder.
  9. Time Series Encoder.
  10. Adapter Design
  11. Sequence Compression
  12. Experiments
  13. Experimental Details
  14. Training Settings
  15. Datasets
  16. ...and 15 more sections

Figures (2)

  • Figure 1: ModRWKV network architecture. Multi-modality data streams undergo initial processing via an encoder, a 1D Convolutional layer, and an adapter. (The 1D Convolutional layer is employed to compress the sequence length of multi-modal inputs, which significantly reduces the computational overhead during training.) Concurrently, text data is transformed through a Text Embedding module. The outputs from the adapter and Text Embedding layers are subsequently concatenated.
  • Figure 2: Performance and efficiency of ModRWKV. Left. The scaling curve of tokens with the performance score. Right. The inference time of ModRWKV with the number of tokens.