Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Shang-Jui Ray Kuo, Paola Cascante-Bonilla

Abstract

Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.

Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Abstract

Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.
Paper Structure (37 sections, 5 equations, 4 figures, 36 tables)

This paper contains 37 sections, 5 equations, 4 figures, 36 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Model Architecture and Notation.
  4. Optimization.
  5. Training
  6. Evaluation Suite.
  7. Investigating Different Vision Encoders
  8. Matched IN1K/224 Backbone Swaps
  9. Dense Objectives Pretrained Backbone Comparisons
  10. Summary of Observations
  11. Analysis, Diagnosis, and Stabilizations
  12. Localization Matters for General VQA
  13. VMamba is Strong Under the Matched IN1K/224 Setting
  14. Dense-Task Objectives Help
  15. Diagnosing Failures
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Overview of our controlled vision encoder study. We follow a LLaVA-style VLM design where an input image is encoded by a frozen vision encoder into visual tokens, which are then processed by the LLM; this modular setup enables controlled swaps of vision encoders from different architecture families under a fixed training recipe. Right: Each marker summarizes one evaluated vision-backbone checkpoint plugged into the same VLM setting. Colors denote the backbone family; marker shapes denote the pretraining objective (classification, detection, or segmentation); marker size reflects encoder scale. Gray markers indicate configurations that exhibit collapse; arrows point to the corresponding stabilized variants after applying our stabilizations.
  • Figure 2: (Top) Grounding examples. Under the matched IN1K/224 setting, VMamba-T (blue) predicts boxes closer to ground truth (green) than ViT-S (red). (Bottom) Token--region similarity. Similarity maps between the visual feature map and the corresponding text tokens from an intermediate LLM layer show VMamba-T yields sharper, more spatially localized text--vision alignment than ViT-S, indicating better preservation and utilization of spatial information.
  • Figure 3: Correlation matrix of benchmark scores across all evaluated models.
  • Figure 4: Scaling of single-GPU VLM inference cost with image resolution at batch size 1. The x-axis shows the square input resolution, written as $R^2$. The six panels report (a) host-side vision latency, (b) host-side LLM latency, (c) GPU vision latency, (d) GPU LLM latency, (e) end-to-end VLM latency, and (f) peak allocated GPU memory. For each model and resolution, we run 100 warm-up iterations and report the mean over the next 100 iterations. Dotted segments ending with an $\times$ indicate the first resolution that exceeds GPU memory capacity. The marker is placed at the top of each subplot to show the out-of-memory threshold rather than a measured metric value. In panel (f), it therefore marks the first failed resolution, not an observed peak-memory measurement.