RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

TL;DR

RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding, and computes query-frame projective coordinates for computing multi-frequency similarity to achieve SE(3) invariance.

Abstract

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet the above desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding. To achieve SE(3) invariance, RayRoPE computes query-frame projective coordinates for computing multi-frequency similarity. Lastly, as the 'predicted' 3D point along a ray may not be precise, RayRoPE presents a mechanism to analytically compute the expected position encoding under uncertainty. We validate RayRoPE on the tasks of novel-view synthesis and stereo depth estimation and show that it consistently improves over alternate position encoding schemes (e.g. 15% relative improvement on LPIPS in CO3D). We also show that RayRoPE can seamlessly incorporate RGB-D input, resulting in even larger gains over alternatives that cannot positionally encode this information.

Figures (12)

• Figure 1: Desiderata for a Multi-View Position Encoding. We seek the following properties for position encoding for multi-view attention: (a) The attention output should be invariant to the choice of global coordinate, namely $SE(3)$ invariance. (b) The positional encoding of tokens that correspond to the same patch observed across different images should be the same. (c) The positional encoding can vary with the underlying scene geometry e.g. allowing a higher similarity when patches see a common 3D point compared to when they don't. (d) Analogous to common 1D and 2D encodings, aspects of the positional encoding should vary at different frequencies, thus allowing a multi-frequency similarity computation.
• Figure 2: Overview of RayRoPE.(a) We encode image patch position as a ray segment $\mathbf{x}=(\mathbf{c},\mathbf{p}^d)$, where $\mathbf{c}$ is the camera center and $\mathbf{p}^d$ is the point at depth $d$ along the ray $r$. We use a linear layer to allow each token to predict the depth $d$ along the ray, thus enabling RayRoPE to adapt to the scene geometry. (b) To ensure $SE(3)$ invariance, we compute the positional encodings using ray positions projected to the query camera frame with $P_i=K_iT_i$, yielding $\tilde{\mathbf{x}}_j=\pi(P_i,\mathbf{x}_j)$. (c) To model the uncertainty in depth prediction, we also predict an uncertainty $\sigma$, yielding an estimated range between $\mathbf{p}^{d-\sigma}$ and $\mathbf{p}^{d+\sigma}$, and use an analytically computed expected position encoding for the corresponding token.
• Figure 3: Applying RayRoPE to the Attention Layer. We apply a linear layer on features $\mathbf{\tau}$ to predict a per-token depth, which is used to compute ray segments. When a query token $\mathbf{\tau}_i$ attends to a set of key tokens $\{\mathbf{\tau}_j\}$, we compute positional encoding $\rho_D$ by projecting rays with query camera $P_i$. We apply the RayRoPE encoding to $\mathbf{q}, \mathbf{k}, \mathbf{v}$ and $\mathbf{o}$ features.
• Figure 4: Qualitative examples on novel view synthesis. For target views with significant overlaps to reference views, RayRoPE produces sharper details than camera-based baseline methods by multi-frequency encodings. For more challenging scenes with large camera variations, RayRoPE also synthesizes more 3D consistent views.
• Figure 5: Qualitative examples on stereo depth estimation. We visualize reprojected 3D points from stereo depth estimation results. When applied to UniMatch unimatch, RayRoPE leads to more accurate depth predictions, resulting in improved 3D reconstruction quality.