Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention

Yuliang Zhan, Xinyu Tang, Han Wan, Jian Li, Ji-Rong Wen, Hao Sun

TL;DR

The paper analyzes how CoT reasoning is encoded across LLMs and VLMs using Linear Artificial Tomography and finds shared low-frequency CoT directions. It introduces L2V-CoT, a training-free latent-intervention that extracts LLM CoT patterns, applies frequency-domain resampling to align dimensions, and injects them into VLMs during inference to boost reasoning. Across diverse benchmarks and VLM architectures, L2V-CoT delivers consistent improvements over training-free baselines and rivals some supervised approaches, with performance scaling alongside injector strength and CoT example count. The method is architecture-agnostic, layer-dependent, and synergistic with explicit reasoning techniques, offering a practical path to transfer high-level reasoning from LLMs to VLMs without retraining.

Abstract

Recently, Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs), but Vision-Language Models (VLMs) still struggle with multi-step reasoning tasks due to limited multimodal reasoning data. To bridge this gap, researchers have explored methods to transfer CoT reasoning from LLMs to VLMs. However, existing approaches either need high training costs or require architectural alignment. In this paper, we use Linear Artificial Tomography (LAT) to empirically show that LLMs and VLMs share similar low-frequency latent representations of CoT reasoning despite architectural differences. Based on this insight, we propose L2V-CoT, a novel training-free latent intervention approach that transfers CoT reasoning from LLMs to VLMs. L2V-CoT extracts and resamples low-frequency CoT representations from LLMs in the frequency domain, enabling dimension matching and latent injection into VLMs during inference to enhance reasoning capabilities. Extensive experiments demonstrate that our approach consistently outperforms training-free baselines and even surpasses supervised methods.

L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention

TL;DR

The paper analyzes how CoT reasoning is encoded across LLMs and VLMs using Linear Artificial Tomography and finds shared low-frequency CoT directions. It introduces L2V-CoT, a training-free latent-intervention that extracts LLM CoT patterns, applies frequency-domain resampling to align dimensions, and injects them into VLMs during inference to boost reasoning. Across diverse benchmarks and VLM architectures, L2V-CoT delivers consistent improvements over training-free baselines and rivals some supervised approaches, with performance scaling alongside injector strength and CoT example count. The method is architecture-agnostic, layer-dependent, and synergistic with explicit reasoning techniques, offering a practical path to transfer high-level reasoning from LLMs to VLMs without retraining.

Abstract

Recently, Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs), but Vision-Language Models (VLMs) still struggle with multi-step reasoning tasks due to limited multimodal reasoning data. To bridge this gap, researchers have explored methods to transfer CoT reasoning from LLMs to VLMs. However, existing approaches either need high training costs or require architectural alignment. In this paper, we use Linear Artificial Tomography (LAT) to empirically show that LLMs and VLMs share similar low-frequency latent representations of CoT reasoning despite architectural differences. Based on this insight, we propose L2V-CoT, a novel training-free latent intervention approach that transfers CoT reasoning from LLMs to VLMs. L2V-CoT extracts and resamples low-frequency CoT representations from LLMs in the frequency domain, enabling dimension matching and latent injection into VLMs during inference to enhance reasoning capabilities. Extensive experiments demonstrate that our approach consistently outperforms training-free baselines and even surpasses supervised methods.

Paper Structure

This paper contains 38 sections, 8 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Chain-of-Thought Reasoning.
  4. Activation engineering.
  5. Preliminary
  6. Linear Artificial Tomography.
  7. Empirical Analysis
  8. Inducing and Capturing Internal Activations.
  9. VLMs and LLMs have similar CoT reasoning encoding patterns.
  10. Low-Frequency CoT direction representations of VLM encode CoT features and activate reasoning ability.
  11. Distribution of Low-Frequency CoT direction representations in VLM and LLM exhibit substantial overlap.
  12. CoT Reasoning Capability transfer via Latent Intervention
  13. Extraction of low-pass CoT pattern representations
  14. Latent Intervention
  15. Experiments
  16. ...and 23 more sections

Figures (12)

  • Figure 1: (a) Distribution of CoT and non-CoT representations in VLMs and LLMs. (b) Effect of low-pass filtering on VLM CoT direction representation. (c) Effect of low-pass filtering on LLM CoT direction representation.
  • Figure 2: (a) Performance of Qwen2VL and its injected variants on MathVista-math and MMStar-reasoning. “HPF” injects high-frequency features. “LPF” injects low-frequency features. (b) The direction representation distribution for VLM and LLM after low-pass filtering(math domain).
  • Figure 3: (a) The overview of Latent Intervention for LLM-to-VLM CoT Transfertion (L2V-CoT). (b) The block of Intervention-Encoder. (c) Low-pass LLM CoT direction representation extraction process. (d) The block of Latent Intervention.
  • Figure 4: Effect on Response Length and Accuracy. (a) InternVL2-8B. (b) Qwen2-VL-7B-Instruct.
  • Figure 5: Qwen2-VL-7B-Instruct accuracy gain on MMStar with L2V-CoT injection at different layers. Y-axis indicates accuracy change over non-CoT responses.
  • ...and 7 more figures