Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Cognitive Supersensing in Multimodal Large Language Model

Boyi Li, Yifan Shen, Yuanzhe Liu, Yifan Xu, Jiateng Liu, Xinzhuo Li, Zhengyuan Li, Jingyuan Zhu, Yunhan Zhong, Fangzhou Lan, Jianguo Cao, James M. Rehg, Heng Ji, Ismini Lourentzou, Xu Cao

TL;DR

This work tackles the limitation of multimodal LLMs in high-level visual cognition by introducing Cognitive Supersensing, which endows models with human-like visual imagery through a Latent Visual Imagery Prediction (LVIP) head and a three-stage training pipeline (Reasoning Chain Generation, LVIP-augmented SFT, and Latent-Rationale RL). The authors also present CogSense-Bench, a comprehensive VQA benchmark spanning five cognitive dimensions to evaluate visual cognition beyond perceptual recognition. Empirical results show that CogSense-8B achieves state-of-the-art performance on CogSense-Bench (平均 73.8%; +33.5 over GPT-5.2) and better out-of-domain generalization on mathematics and science VQA tasks, while preserving strong general VLM capabilities. These findings suggest that internal visual imagery can effectively bridge perceptual recognition and higher-order cognitive reasoning, with open-source CogSense-Bench and model weights for further research.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in open-vocabulary perceptual tasks, yet their ability to solve complex cognitive problems remains limited, especially when visual details are abstract and require visual memory. Current approaches primarily scale Chain-of-Thought (CoT) reasoning in the text space, even when language alone is insufficient for clear and structured reasoning, and largely neglect visual reasoning mechanisms analogous to the human visuospatial sketchpad and visual imagery. To mitigate this deficiency, we introduce Cognitive Supersensing, a novel training paradigm that endows MLLMs with human-like visual imagery capabilities by integrating a Latent Visual Imagery Prediction (LVIP) head that jointly learns sequences of visual cognitive latent embeddings and aligns them with the answer, thereby forming vision-based internal reasoning chains. We further introduce a reinforcement learning stage that optimizes text reasoning paths based on this grounded visual latent. To evaluate the cognitive capabilities of MLLMs, we present CogSense-Bench, a comprehensive visual question answering (VQA) benchmark assessing five cognitive dimensions. Extensive experiments demonstrate that MLLMs trained with Cognitive Supersensing significantly outperform state-of-the-art baselines on CogSense-Bench and exhibit superior generalization on out-of-domain mathematics and science VQA benchmarks, suggesting that internal visual imagery is potentially key to bridging the gap between perceptual recognition and cognitive understanding. We will open-source the CogSense-Bench and our model weights.

Toward Cognitive Supersensing in Multimodal Large Language Model

TL;DR

This work tackles the limitation of multimodal LLMs in high-level visual cognition by introducing Cognitive Supersensing, which endows models with human-like visual imagery through a Latent Visual Imagery Prediction (LVIP) head and a three-stage training pipeline (Reasoning Chain Generation, LVIP-augmented SFT, and Latent-Rationale RL). The authors also present CogSense-Bench, a comprehensive VQA benchmark spanning five cognitive dimensions to evaluate visual cognition beyond perceptual recognition. Empirical results show that CogSense-8B achieves state-of-the-art performance on CogSense-Bench (平均 73.8%; +33.5 over GPT-5.2) and better out-of-domain generalization on mathematics and science VQA tasks, while preserving strong general VLM capabilities. These findings suggest that internal visual imagery can effectively bridge perceptual recognition and higher-order cognitive reasoning, with open-source CogSense-Bench and model weights for further research.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in open-vocabulary perceptual tasks, yet their ability to solve complex cognitive problems remains limited, especially when visual details are abstract and require visual memory. Current approaches primarily scale Chain-of-Thought (CoT) reasoning in the text space, even when language alone is insufficient for clear and structured reasoning, and largely neglect visual reasoning mechanisms analogous to the human visuospatial sketchpad and visual imagery. To mitigate this deficiency, we introduce Cognitive Supersensing, a novel training paradigm that endows MLLMs with human-like visual imagery capabilities by integrating a Latent Visual Imagery Prediction (LVIP) head that jointly learns sequences of visual cognitive latent embeddings and aligns them with the answer, thereby forming vision-based internal reasoning chains. We further introduce a reinforcement learning stage that optimizes text reasoning paths based on this grounded visual latent. To evaluate the cognitive capabilities of MLLMs, we present CogSense-Bench, a comprehensive visual question answering (VQA) benchmark assessing five cognitive dimensions. Extensive experiments demonstrate that MLLMs trained with Cognitive Supersensing significantly outperform state-of-the-art baselines on CogSense-Bench and exhibit superior generalization on out-of-domain mathematics and science VQA benchmarks, suggesting that internal visual imagery is potentially key to bridging the gap between perceptual recognition and cognitive understanding. We will open-source the CogSense-Bench and our model weights.
Paper Structure (28 sections, 11 equations, 7 figures, 6 tables)

This paper contains 28 sections, 11 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. CogSense Dataset and Benchmark
  3. Method
  4. Preliminaries
  5. Stage I: Reasoning Chain Generation
  6. Stage II: SFT with Latent Visual Imagery Prediction
  7. Stage III: RL with Latent Rationales
  8. Experiments
  9. Experimental Settings
  10. Cognitive Ability Results
  11. General Ability Results
  12. Ablation Study
  13. Out-of-Domain Evaluation
  14. Related Work
  15. Conclusion
  16. ...and 13 more sections

Figures (7)

  • Figure 1: CogSense-Dataset Examples. Samples across each category from the CogSense-Dataset. CogSense-Dataset comprises various visual cognitive questions classified into five categories: Fluid Intelligence, Crystallized Intelligence, Visuospatial Cognition, Mental Simulation, and Visual Routines, which require visual imagery and cognitive supersensing with deep thinking and reasoning.
  • Figure 2: CogSense-Dataset Distribution. The data distribution of our CogSense-Dataset-105K.
  • Figure 3: The framework of Cognitive Surpersensing.Left:Architecture Overview. CogSense-8B is a VLM that takes images and prompts as input with a text decoder to generate the answer and a Latent Visual Imagery Prediction (LVIP) head to generate a latent visual imagery of the option-image in parallel. Right:Method Overview. To train CogSense-8B, we (1) generate reasoning paths via LLMs, (2) implement SFT to jointly optimize the LVIP head and the model weights, and (3) implement RL to further optimize reasoning paths with Latent Rationales.
  • Figure 4: Qualitative Example of Visual Cognition Reasoning Across Models. We underline decisive sentences in the reasoning chain. CogSense-8B demonstrates a coherent, multi-step logical chain that closely matches the ground truth, while other models exhibit less precise or less interpretable reasoning paths
  • Figure 5: EMMA Benchmark Sample Problems.
  • ...and 2 more figures