MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing

TL;DR

MambaEye introduces a truly size-agnostic visual encoder by reframing image classification as a causal, sequential task processed by a Mamba2 backbone. It couples a relative move embedding to encode spatial shifts across arbitrary patch sequences with a diffusion-inspired loss that provides dense, step-wise supervision as evidence accumulates. Empirically, it achieves competitive ImageNet-1K performance across a range of resolutions and shows robust scaling with sequence length, outperforming prior Mamba-based encoders at high resolutions. The work demonstrates that unidirectional, memory-efficient sequential vision can match static architectures while offering dynamic inference and resolution flexibility, with promising avenues for extensions to video and 3D data.

Abstract

Despite decades of progress, a truly input-size agnostic visual encoder-a fundamental characteristic of human vision-has remained elusive. We address this limitation by proposing \textbf{MambaEye}, a novel, causal sequential encoder that leverages the low complexity and causal-process based pure Mamba2 backbone. Unlike previous Mamba-based vision encoders that often employ bidirectional processing, our strictly unidirectional approach preserves the inherent causality of State Space Models, enabling the model to generate a prediction at any point in its input sequence. A core innovation is our use of relative move embedding, which encodes the spatial shift between consecutive patches, providing a strong inductive bias for translation invariance and making the model inherently adaptable to arbitrary image resolutions and scanning patterns. To achieve this, we introduce a novel diffusion-inspired loss function that provides dense, step-wise supervision, training the model to build confidence as it gathers more visual evidence. We demonstrate that MambaEye exhibits robust performance across a wide range of image resolutions, especially at higher resolutions such as $1536^2$ on the ImageNet-1K classification task. This feat is achieved while maintaining linear time and memory complexity relative to the number of patches.

Figures (5)

• Figure 1: MambaEye Resolution Scaling Our MambaEye-S (11M params) models are benchmarked against Mamba-based models of similar size. While deterministic scanning methods like FractalMamba++ li2025scaling achieve higher peak accuracy at medium resolutions, our model demonstrates superior scaling at extreme resolutions. Notably, MambaEye outperforms FractalMamba++ at resolutions over $1280^2$, despite using only a naive random sampling policy. This highlights our architecture's inherent robustness and size-agnostic capabilities.
• Figure 2: Illustration of the Sequential Data Processing Pipeline. The figure shows four snapshots of the data generation process. An original image is augmented and placed on a padded canvas (outlined by the white frame). At each step $t$, a new $16 \times 16$ patch (red border) is randomly sampled. The cumulative area covered by all patches is colored gray, and its proportion to the total image area defines the information ratio ($r_t$). A move embedding, $\mathbf{m}_t$, is also computed from the relative coordinate shift from the previous patch, providing spatial context to the model.
• Figure 3: MambaEye Model Architecture. The diagram illustrates the sequential flow of the model. At each step, a flattened image patch vector and a sinusoidal "Move Embedding" are concatenated and processed by a projection layer. The core of the model is a stack of 24 Mamba2 blocks, whose output is passed to a final MLP head to produce a classification logit for that step.
• Figure 4: MambaEye performance vs. sequence length ($T$). (a) shows all model variants accumulating information at a fixed $512^2$ resolution. Accuracy saturates for all models, with fine-tuned models achieving higher performance. (b) shows the MambaEye-B (FT) model across different resolutions, illustrating the trade-off between image size and required sequence length. At lower resolutions, performance saturates quickly, while higher resolutions require more steps.
• Figure 5: Deterministic scanning patterns: performance vs. sequence length. Top-1 accuracy of the MambaEye-B (FT) model under horizontal raster and horizontal zigzag scans, plotted over the sequence length ($T=4096$) for multiple resolutions. For the raster scan, wrapping to the first pixel of the next row causes large horizontal jumps and pronounced high-frequency oscillations; for the zigzag scan, the inter-row transition is a 1-pixel vertical move with adjacent horizontal continuity, suppressing high-frequency oscillations but this method still performs substantially worse compared to random sampling.