Towards Visuospatial Cognition via Hierarchical Fusion of Visual Experts

TL;DR

The paper tackles the gap in visuospatial cognition within multimodal large language models by introducing ViCA2, a compact dual-encoder architecture combining SigLIP for global semantics and Hiera for fine-grained spatial structure, guided by a token-ratio mechanism for efficiency. It also releases ViCA-322K, a large spatially grounded QA dataset, and demonstrates state-of-the-art VSI-Bench performance for a 7B model, outperforming larger open-source and some proprietary models. The approach shows that modular vision backbones and targeted spatial data can yield strong spatial reasoning without enormous parameter counts, while highlighting areas for improvement such as relative direction understanding. The work provides open-source code, model weights, and a substantial dataset to catalyze further research into visuospatial cognition in multimodal systems.

Abstract

While Multimodal Large Language Models (MLLMs) excel at general vision-language tasks, visuospatial cognition - reasoning about spatial layouts, relations, and dynamics - remains a significant challenge. Existing models often lack the necessary architectural components and specialized training data for fine-grained spatial understanding. We introduce ViCA2 (Visuospatial Cognitive Assistant 2), a novel MLLM designed to enhance spatial reasoning. ViCA2 features a dual vision encoder architecture integrating SigLIP for semantics and Hiera for spatial structure, coupled with a token ratio control mechanism for efficiency. We also developed ViCA-322K, a new large-scale dataset with over 322,000 spatially grounded question-answer pairs for targeted instruction tuning. On the challenging VSI-Bench benchmark, our ViCA2-7B model achieves a state-of-the-art average score of 56.8, significantly surpassing larger open-source models (e.g., LLaVA-NeXT-Video-72B, 40.9) and leading proprietary models (Gemini-1.5 Pro, 45.4). This demonstrates the effectiveness of our approach in achieving strong visuospatial intelligence with a compact model. We release ViCA2, its codebase, and the ViCA-322K dataset to facilitate further research.

Figures (25)

• Figure 1: Average performance comparison on VSI-Bench. We compare ViCA2-7B with both proprietary API-based models and leading open-source multimodal models on the VSI-Bench benchmark, which evaluates visuospatial reasoning across 8 tasks. Despite having only 7B parameters, ViCA2-7B significantly outperforms all other models, achieving an average score of 56.8, surpassing the best-performing proprietary model (Gemini-1.5 Proteam2024gemini, 45.4) and the strongest open-source baseline (LLaVA-NeXT-Video-72B, 40.9) by a large margin. This highlights the effectiveness of our dual vision encoder design and hierarchical fusion approach for spatial understanding.
• Figure 2: An overview of the ViCA2 architecture. The model integrates SigLIP for global semantic understanding and Hiera for spatial structure encoding, with a token ratio control mechanism to balance expressiveness and memory efficiency. A complete version of the architecture diagram is provided in the Appendix(Figure \ref{['fig:vica2_arch_complete']}).
• Figure 3: Average gpt-4.1-mini scores (0--10) for in-detail descriptions generated by each of the four checkpoints on VSI-Bench.
• Figure 4: Impact of training data size on ViCA2-7B performance on VSI-Bench. The average score consistently improves as the percentage of ViCA-322K training data increases, demonstrating that ViCA2-7B benefits from larger-scale spatially grounded instruction tuning and has not yet reached saturation with the current 322K samples.
• Figure 6: An overview of the ViCA2 architecture. It integrates SigLIP for global semantics and Hiera for spatial structure, with token ratio control to balance expressiveness and memory efficiency.