RayPose: Ray Bundling Diffusion for Template Views in Unseen 6D Object Pose Estimation

TL;DR

This work presents RayPose, a diffusion-transformer framework for unseen 6D object pose estimation that treats pose inference as ray-bundle alignment between a query image and multiple posed templates. By adopting an object-centered ray representation for rotation and a dense offset map for translation, the model decouples pose components and leverages multiview priors through a conditional diffusion process. A coarse-to-fine training strategy enables robust pose reasoning without architectural changes, while cross-attention in the diffusion transformer enables effective fusion of query and template information. Experiments on multiple BOP datasets show competitive or superior performance for RGB-only unseen-object pose estimation, with ablations validating the importance of template priors, multiview conditioning, and the proposed loss terms.

Abstract

Typical template-based object pose pipelines estimate the pose by retrieving the closest matching template and aligning it with the observed image. However, failure to retrieve the correct template often leads to inaccurate pose predictions. To address this, we reformulate template-based object pose estimation as a ray alignment problem, where the viewing directions from multiple posed template images are learned to align with a non-posed query image. Inspired by recent progress in diffusion-based camera pose estimation, we embed this formulation into a diffusion transformer architecture that aligns a query image with a set of posed templates. We reparameterize object rotation using object-centered camera rays and model object translation by extending scale-invariant translation estimation to dense translation offsets. Our model leverages geometric priors from the templates to guide accurate query pose inference. A coarse-to-fine training strategy based on narrowed template sampling improves performance without modifying the network architecture. Extensive experiments across multiple benchmark datasets show competitive results of our method compared to state-of-the-art approaches in unseen object pose estimation.

Figures (3)

• Figure 1: Given a novel object query image, our method accurately predicts the object's 6D pose using a multiview diffusion model conditioned on a set of template images with known poses. Leveraging our proposed structured 2D pose maps, represented as bundles of rays, the diffusion model recovers the query object's pose by progressively denoising these ray bundles.
• Figure 2: Pipeline overview of our method. We represent the 6D object pose using structured rotation and translation maps and employ a diffusion model to estimate the pose from random inputs. Given a query image of an unseen object and multiple template images with known object poses, our method first extracts query embeddings from a query encoder and multiview posed template embeddings from a template encoder. These embeddings serve as conditioning inputs for a diffusion transformer decoder, which is trained to denoise the object pose from random inputs. The model predicts the relative pose between the query and templates, from which the absolute 6D pose of the query object is reconstructed based on the known poses for the templates.
• Figure 3: Visual illustration of the object-centric ray representation used for rotation prediction in our diffusion model. The rotation map ${\mathcal{M}}_R$ is defined as a bundle of rays originating from the object center ${\mathcal{O}}_{obj}$, encoded as a 3-channel 2D map.