Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models

Ruihan Xu, Yuting Gao, Lan Wang, Jianing Li, Weihao Chen, Qingpei Guo, Ming Yang, Shiliang Zhang

TL;DR

This work addresses the underutilization of large multimodal models by reusing existing parameters through recursive refinement. It introduces RecursiveVLM, a recursive Transformer with a Recursive Connector that aligns cross-step features via multi-layer, modality-specific fusion, and a Monotonic Recursion Loss that enforces non-decreasing performance with recursion depth. The key contributions are the RMSNorm-based feature normalization, modality-specific projection with a learnable residual scale, and a training objective that guarantees stable improvements across steps, enabling deployment-adaptive inference. Empirically, RecursiveVLM outperforms standard Transformers and vanilla recursive baselines across eight multimodal benchmarks, with notable gains at shallow recursion and continued improvements as recursion deepens, plus evidence suggesting recursion can mitigate hallucinations. The approach offers a practical path toward efficient, adaptable, and trustworthy multimodal reasoning without increasing model size.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in vision-language tasks, yet their vast parameter counts are often underutilized during both training and inference. In this work, we embrace the idea of looping back to move forward: reusing model parameters through recursive refinement to extract stronger multimodal representations without increasing model size. We propose RecursiveVLM, a recursive Transformer architecture tailored for LMMs. Two key innovations enable effective looping: (i) a Recursive Connector that aligns features across recursion steps by fusing intermediate-layer hidden states and applying modality-specific projections, respecting the distinct statistical structures of vision and language tokens; (ii) a Monotonic Recursion Loss that supervises every step and guarantees performance improves monotonically with recursion depth. This design transforms recursion into an on-demand refinement mechanism: delivering strong results with few loops on resource-constrained devices and progressively improving outputs when more computation resources are available. Experiments show consistent gains of +3% over standard Transformers and +7% over vanilla recursive baselines, demonstrating that strategic looping is a powerful path toward efficient, deployment-adaptive LMMs.

Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models

TL;DR

This work addresses the underutilization of large multimodal models by reusing existing parameters through recursive refinement. It introduces RecursiveVLM, a recursive Transformer with a Recursive Connector that aligns cross-step features via multi-layer, modality-specific fusion, and a Monotonic Recursion Loss that enforces non-decreasing performance with recursion depth. The key contributions are the RMSNorm-based feature normalization, modality-specific projection with a learnable residual scale, and a training objective that guarantees stable improvements across steps, enabling deployment-adaptive inference. Empirically, RecursiveVLM outperforms standard Transformers and vanilla recursive baselines across eight multimodal benchmarks, with notable gains at shallow recursion and continued improvements as recursion deepens, plus evidence suggesting recursion can mitigate hallucinations. The approach offers a practical path toward efficient, adaptable, and trustworthy multimodal reasoning without increasing model size.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in vision-language tasks, yet their vast parameter counts are often underutilized during both training and inference. In this work, we embrace the idea of looping back to move forward: reusing model parameters through recursive refinement to extract stronger multimodal representations without increasing model size. We propose RecursiveVLM, a recursive Transformer architecture tailored for LMMs. Two key innovations enable effective looping: (i) a Recursive Connector that aligns features across recursion steps by fusing intermediate-layer hidden states and applying modality-specific projections, respecting the distinct statistical structures of vision and language tokens; (ii) a Monotonic Recursion Loss that supervises every step and guarantees performance improves monotonically with recursion depth. This design transforms recursion into an on-demand refinement mechanism: delivering strong results with few loops on resource-constrained devices and progressively improving outputs when more computation resources are available. Experiments show consistent gains of +3% over standard Transformers and +7% over vanilla recursive baselines, demonstrating that strategic looping is a powerful path toward efficient, deployment-adaptive LMMs.
Paper Structure (22 sections, 11 equations, 3 figures, 11 tables)

This paper contains 22 sections, 11 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Multimodal Models
  4. Recursive Transformers
  5. Method
  6. Preliminary
  7. Recursive Transformer
  8. Recursive Connector
  9. Monotonic Recursion Loss
  10. Experiments
  11. Experimental Setup
  12. Implementation Details.
  13. Evaluation Benchmarks.
  14. Main Results
  15. Ablation Studies
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Statistical results in Ming-Lite-Omni ming_omni. (a) The L2 norm of the hidden states in each layer indicates misalignment in feature scale. (b) The Centered Kernel Alignment (CKA) between the hidden states and the input embeddings indicates a feature distribution misalignment in vanilla recursive models.
  • Figure 2: Comparison of recursive architectures for LMMs. (a) Standard non-recursive LMM with single-pass. (b) Vanilla recursion: output embeddings are directly fed back as inputs across steps, causing feature scale and distribution misalignment. (c) Our method fuses features from multiple layers and employs a modality-specific Recursive Connector to bridge recursion steps and mitigate misalignment.
  • Figure 3: The framework of our proposed method. (a) RecursiveVLM loops the LLM backbone with a modality-specific multi-layer connector. (b) Detailed structure of the connector module comprising an RMSNorm, a learnable residual scaling vector, and an MLP.