MUT3R: Motion-aware Updating Transformer for Dynamic 3D Reconstruction

TL;DR

MUT3R addresses motion-induced artifacts in dynamic 3D reconstruction by harnessing motion cues embedded in the attention maps of a frozen CUT3R transformer. It introduces a training-free attention-level gating mechanism that attenuates dynamic content in early decoder layers, preserving the model's geometric reasoning while reducing motion propagation. The approach relies on aggregating self-attention responses across layers to form a patch-wise dynamic score, which biases attention in self- and cross-attention directions. Across video depth, camera pose, and 4D reconstruction benchmarks, MUT3R yields improved temporal stability, pose robustness, and 4D coherence without any architecture changes or fine-tuning, highlighting the potential of attention-derived motion cues for dynamic 3D vision.

Abstract

Recent stateful recurrent neural networks have achieved remarkable progress on static 3D reconstruction but remain vulnerable to motion-induced artifacts, where non-rigid regions corrupt attention propagation between the spatial memory and image feature. By analyzing the internal behaviors of the state and image token updating mechanism, we find that aggregating self-attention maps across layers reveals a consistent pattern: dynamic regions are naturally down-weighted, exposing an implicit motion cue that the pretrained transformer already encodes but never explicitly uses. Motivated by this observation, we introduce MUT3R, a training-free framework that applies the attention-derived motion cue to suppress dynamic content in the early layers of the transformer during inference. Our attention-level gating module suppresses the influence of dynamic regions before their artifacts propagate through the feature hierarchy. Notably, we do not retrain or fine-tune the model; we let the pretrained transformer diagnose its own motion cues and correct itself. This early regulation stabilizes geometric reasoning in streaming scenarios and leads to improvements in temporal consistency and camera pose robustness across multiple dynamic benchmarks, offering a simple and training-free pathway toward motion-aware streaming reconstruction.

Figures (6)

• Figure 1: We present MUT3R, a training-free framework that derives motion cues from CUT3R’s attention maps and performs early-layer motion suppression to enhance robustness in dynamic scenes; unlike pairwise DUSt3R, which requires extra temporal fusion, the streaming design of CUT3R carries historical context in its state tokens, making such dynamics directly reflected in self-attention. (SA denotes self-attention and CA denotes cross-attention in the figure.)
• Figure 2: Method pipeline. Given a sequence of input frames, each image is first encoded into visual tokens $F_t$ by a frozen ViT encoder. The tokens are processed by the recurrent CUT3R decoder with a persistent state. Inside each decoder layer, self-attention (SA) and cross-attention (CA) jointly refine the image and state tokens. We analyze the attention responses from multiple layers to estimate a patch-level dynamic score map. These scores guide our attention-level gating module that suppresses dynamic regions during early-layer attention. The resulting motion-aware updating enables robust and consistence 3D reconstruction without finetuning the base model.
• Figure 3: Visualization of different level attention maps. From left to right, we show the input RGB frame, self-attention responses from shallow layer and deep decoder layer, and the averaged attention map across all layers. Blue indicates lower attention responses, while red indicates higher responses.
• Figure 4: Motion awareness attention gating. This figure illustrates how the dynamic score map $g_t$ is converted into attention biases that suppress motion-dominated regions. The left part shows self-attention gating. The right part visualizes the bias terms for both directions of cross-attention between image and state tokens. The picture depicts the gated update applied to the image tokens, highlighting where our motion-aware suppression reduces state-induced changes in dynamic regions.
• Figure 5: Layer-wise PCA visualization of decoder embeddings before and after early-layer suppression. We visualize the evolution of image-token embeddings across different decoder layers, where $\text{L}k$ represents the embeddings after passing through the $k$-th decoder layer.