FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models

TL;DR

FRISM addresses the challenge of injecting reasoning from LRMs into VLMs without sacrificing visual perception. By performing fine-grained subspace-level merging via SVD on the LRM task vector and learning per-subspace gates, FRISM selectively injects reasoning capabilities while preserving vision, guided by a label-free self-distillation objective and a spectral injection term. Theoretical analysis frames FRISM as a curvature-aware filter that amplifies low-curvature, reasoning-dominated subspaces, and extensive experiments show state-of-the-art or competitive gains across multiple VL reasoning benchmarks and diverse model types, with improved efficiency through low-rank variants. The approach offers a robust, plug-and-play path to enhance VL reasoning in practical settings, reducing reliance on costly multimodal reasoning data and post-training finetuning.

Abstract

Efficiently enhancing the reasoning capabilities of Vision-Language Models (VLMs) by merging them with Large Reasoning Models (LRMs) has emerged as a promising direction. However, existing methods typically operate at a coarse-grained layer level, which often leads to a trade-off between injecting reasoning capabilities and preserving visual capabilities. To address this limitation, we propose {FRISM} (Fine-grained Reasoning Injection via Subspace-level model Merging), a fine-grained reasoning injection framework based on subspace-level model merging. Observing that reasoning capabilities are encoded in distinct subspaces, FRISM decomposes LRM task vectors via Singular Value Decomposition (SVD) and adaptively tunes the scaling coefficients of each subspace through learning to realize fine-grained reasoning injection. Furthermore, we introduce a label-free self-distillation learning strategy with a dual-objective optimization using common vision-language perception datasets. Extensive experiments demonstrate that FRISM effectively improves reasoning capabilities without compromising the model's original visual capabilities by consistently achieving state-of-the-art performance across diverse visual reasoning benchmarks.

Figures (8)

• Figure 1: Illustration of different model merging strategies.
• Figure 2: Vision-Reasoning tradeoff when merging VLMs and LRMs. Task Arithmetic and IP-Merging are applied under different merging coefficients and similarity thresholds, respectively.
• Figure 3: Impact of scaling coefficients across subspace ranks when merging Qwen2.5-VL-7B-Instruct and the subspaces of DeepSeek-R1-Distill-Qwen-7B task vector.
• Figure 4: FRISM Framework. (a) The overall framework. FRISM transfers the reasoning capabilities from LRMs to VLMs. FRISM can be divided into two stages. (b) Stage 1: Decomposition and Initialization. (c) Stage 2: Injection and Training.
• Figure 5: Count of self-reflection tokens on MathVision dataset of different merging methods.