Large Vision Models Can Solve Mental Rotation Problems

TL;DR

The paper investigates whether large vision models develop mental-rotation-like abilities by conducting a layer-wise evaluation of ViT, CLIP, DINOv2, and DINOv3 across Shepard-Metzler-style blocks, text variants, and photo-realistic scenes. It employs a Siamese classifier on layer embeddings to assess rotation-equivalence versus mirroring, across all transformer layers and task variants. Key findings show that self-supervised models (CLIP, DINOv2, DINOv3) better preserve geometric structure than supervised ViTs, with intermediate layers carrying the most pose information and performance deteriorating as rotation complexity or occlusion increases; DINOv3Huge can solve the hardest tasks. These results imply that training objectives promoting geometric sensitivity and preserving pose cues across layers are crucial for enabling robust geometric reasoning in vision systems.

Abstract

Mental rotation is a key test of spatial reasoning in humans and has been central to understanding how perception supports cognition. Despite the success of modern vision transformers, it is still unclear how well these models develop similar abilities. In this work, we present a systematic evaluation of ViT, CLIP, DINOv2, and DINOv3 across a range of mental-rotation tasks, from simple block structures similar to those used by Shepard and Metzler to study human cognition, to more complex block figures, three types of text, and photo-realistic objects. By probing model representations layer by layer, we examine where and how these networks succeed. We find that i) self-supervised ViTs capture geometric structure better than supervised ViTs; ii) intermediate layers perform better than final layers; iii) task difficulty increases with rotation complexity and occlusion, mirroring human reaction times and suggesting similar constraints in embedding space representations.

Figures (5)

• Figure 1: Overview of the experimental pipeline for the mental rotation task. Pairs of objects are rendered from different viewpoints and passed through a shared Vision Transformer (ViT). The resulting embeddings are compared in a Siamese network to determine whether the two views correspond to the same object under rotation or to a mirrored counterpart. This setup tests whether model representations preserve pose information sufficient to solve mental rotation problems.
• Figure 2: Overview of the seven dataset variants used in the experiments. Shepard-Metzler tasks include pairs with small relative elevation rotations (±0°) and unconstrained rotations (Free). Text tasks include natural words (Normal), randomly sampled character strings (Random), and artificial symbols rendered in the PseudoSloan font (Pseudo). Photo-Realistic tasks consist of tabletop object scenes captured from two viewpoints, with either a 90° or 30° camera azimuth angle.
• Figure 3: Test accuracy across layers for four model families (ViT, CLIP, DINOv2, DINOv3) in three sizes (Base, Large, Huge). Results are averaged over three runs of stratified 10-fold cross-validation, with shaded regions indicating standard error. Text-related tasks are preserved in the final layers only for CLIP and DINOv3, whereas all models retain text-related information in earlier layers. Photo-Realistic scenes and Shepard-Metzler ±0° objects are consistently well-represented in early and middle layers, but most models lose this signal toward the final layers, except for DINOv3. For Shepard-Metzler Free objects, only the deepest layers of DINOv3Huge (layers 18–19) maintain discriminative information.
• Figure 4: Test accuracy across layers for DINOv2Huge and DINOv3Huge on extended Shepard-Metzler (top) and Photo-Realistic (bottom) datasets. Shepard-Metzler tasks vary in relative rotation angle (±0° to Free), and Photo-Realistic tasks in camera elevation (30°–90°). Accuracy declines as transformations become harder, with both models strongest in early/mid layers for simple rotations and weakest for large pose changes or unconstrained rotations.
• Figure 5: Projection onto the first two principal components of DINOv2Huge and DINOv3Huge for layer 3 (when the model cannot yet solve the task) and layer 16 (when it can). For the simple Shepard-Metzler task (±0°) with blocks rotated incrementally over 360°, the later layer shows a clear, continuous structure aligned with rotation angle, unlike the early layer.