RFM-Pose:Reinforcement-Guided Flow Matching for Fast Category-Level 6D Pose Estimation

TL;DR

RFM-Pose tackles category-level 6D pose estimation under rotational symmetry by marrying fast flow-matching generation with reinforcement-learning–driven refinement and value-guided hypothesis scoring. It trains a conditional flow-matching backbone to map a Gaussian prior to the pose distribution on SE(3) and then uses proximal policy optimization with a multi-critic value network to refine the sampling trajectory toward geometrically accurate poses, followed by value-based aggregation over multiple candidates. The approach yields a strong accuracy-efficiency balance on REAL275, CAMERA, and Omni6DPose, achieving competitive or superior results with far fewer sampling steps and enabling real-time pose tracking. This unified framework offers practical benefits for real-world robotics and AR tasks by reducing computation while maintaining robust, multimodal pose estimates, and it sets the stage for incorporating additional cues such as RGB information in future work.

Abstract

Object pose estimation is a fundamental problem in computer vision and plays a critical role in virtual reality and embodied intelligence, where agents must understand and interact with objects in 3D space. Recently, score based generative models have to some extent solved the rotational symmetry ambiguity problem in category level pose estimation, but their efficiency remains limited by the high sampling cost of score-based diffusion. In this work, we propose a new framework, RFM-Pose, that accelerates category-level 6D object pose generation while actively evaluating sampled hypotheses. To improve sampling efficiency, we adopt a flow-matching generative model and generate pose candidates along an optimal transport path from a simple prior to the pose distribution. To further refine these candidates, we cast the flow-matching sampling process as a Markov decision process and apply proximal policy optimization to fine-tune the sampling policy. In particular, we interpret the flow field as a learnable policy and map an estimator to a value network, enabling joint optimization of pose generation and hypothesis scoring within a reinforcement learning framework. Experiments on the REAL275 benchmark demonstrate that RFM-Pose achieves favorable performance while significantly reducing computational cost. Moreover, similar to prior work, our approach can be readily adapted to object pose tracking and attains competitive results in this setting.

Figures (8)

• Figure 1: Speed-accuracy trade-off comparison on REAL275. Our method achieves favorable balance between inference speed and pose accuracy.
• Figure 2: Overview of the RFM-Pose framework consisting of three stages. Left: Flow-matching model training using conditional flow-matching objective with PointNet++ feature extraction. Right Top: PPO-based refinement where the pretrained flow model serves as initial policy, generating pose candidates and optimizing policy and value networks with rotation/translation error rewards. Right Bottom: Inference stage where $K$ candidates are generated, ranked by value network(critic), and aggregated via QUEST (rotation) and weighted averaging (translation) to obtain final pose estimate.
• Figure 3: Network architecture of the policy and multi-critic value network. The model processes pose at timestep $t$, timestep embedding, and partial observation point cloud.
• Figure 4: Per-category quantitative comparison with GenPose zhang2024generative on REAL275. The left represents GenPose, while the right represents our approach.
• Figure 5: Qualitative comparison between our method and GenPose on six representative scenes from the REAL275 test set. Left is our method, right is the baseline method for reference