C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning

TL;DR

RoPE's one-dimensional raster-scan indexing disrupts 2D visual continuity and its long-range decay causes image token neglect in 3D LMMs. The authors propose C$^{2}$RoPE, combining a spatio-temporal continuous positional embedding that forms a triplet index $(m,x,y)$ with a frequency allocation across components and a Chebyshev Causal Masking based on 2D distance from the image center. Evaluations on ScanQA and SQA3D show consistent improvements over the LLaVA-3D baseline and other 3D LMMs, including EM@1 gains of +4.3 on ScanQA and +1.2 on SQA3D, indicating enhanced spatial reasoning and token utilization. The approach offers a practical enhancement for 3D multimodal reasoning and is accompanied by public code at the provided repository.

Abstract

Recent advances in 3D Large Multimodal Models (LMMs) built on Large Language Models (LLMs) have established the alignment of 3D visual features with LLM representations as the dominant paradigm. However, the inherited Rotary Position Embedding (RoPE) introduces limitations for multimodal processing. Specifically, applying 1D temporal positional indices disrupts the continuity of visual features along the column dimension, resulting in spatial locality loss. Moreover, RoPE follows the prior that temporally closer image tokens are more causally related, leading to long-term decay in attention allocation and causing the model to progressively neglect earlier visual tokens as the sequence length increases. To address these issues, we propose C^2RoPE, an improved RoPE that explicitly models local spatial Continuity and spatial Causal relationships for visual processing. C^2RoPE introduces a spatio-temporal continuous positional embedding mechanism for visual tokens. It first integrates 1D temporal positions with Cartesian-based spatial coordinates to construct a triplet hybrid positional index, and then employs a frequency allocation strategy to encode spatio-temporal positional information across the three index components. Additionally, we introduce Chebyshev Causal Masking, which determines causal dependencies by computing the Chebyshev distance of image tokens in 2D space. Evaluation results across various benchmarks, including 3D scene reasoning and 3D visual question answering, demonstrate C^2RoPE's effectiveness. The code is be available at https://github.com/ErikZ719/C2RoPE.

Figures (5)

• Figure 1: (a) illustrates RoPE’s raster-scan positional index assignment; (b) visualizes the information flow from image tokens to instruction tokens, where flows from each image token are aggregated and arranged according to their positions in 2D space. The statistics are averaged over a 3K VQA subset. (c) shows the quantitative analysis of information flow from visual tokens to output tokens.
• Figure 2: Structure of $C^2$RoPE. We integrate temporal indices and Cartesian spatial coordinates to form a triplet hybrid positional index that preserves local continuity of visual tokens. A frequency-based allocation encodes complementary spatio-temporal cues across the triplet, and Chebyshev Causal Masking enforces locality-aware causality by modulating attention with the 2D Chebyshev distance. This design mitigates long-range attention decay and improves 3D scene reasoning in LMMs.
• Figure 3: Long-distance position information attenuation of llava-3D, (a) represents the long-distance information attenuation problem in LLaVA, (b) (c) represent CCA-LLaVA cca-llava and MCA-LLaVA mca to alleviate the decay of long-distance information flow by optimizing ROPE, (d) represents our method.
• Figure 4: The position embeddings of adjacent tokens for Vanilla RoPE (left), frame‑aligned visual tokens for M‑RoPE (center‑left), interleaved spatio‑temporal tokens for VideoRoPE (center‑right), and our $C^2$RoPE (right) with triplet hybrid indexing and Chebyshev causal masking.
• Figure 5: The case study is conducted on samples from SQA3D, where we provide identical model inputs to examine the generated responses.