Monocular Visual 8D Pose Estimation for Articulated Bicycles and Cyclists

TL;DR

This work tackles the inadequacy of rigid 6D pose models for articulated bicycles by introducing monocular category-level 8D pose estimation from a single RGB image. The approach uses a two-branch network: a Keypoint Branch with a ViT-based encoder to predict the bicycle's rotations $\theta_X,\theta_Y,\theta_Z$, pedal/steering angles $\theta_p,\theta_s$, and 3D keypoint residuals, and a Translation Branch to regress the 3D translation $\mathbf{T}=(t_X,t_Y,t_Z)$, both aided by a parametric 3D bicycle model that reposes canonical keypoints $\kappa_{c3D}=\kappa_{\mu}+\Delta_{\kappa}$ and projects them with camera matrix $P$ for supervision. The model is trained with a hybrid dataset consisting of synthetic 3D data (57{,}500 instances across 23 bicycles) and real 2D keypoint data from Waymo/COCO, employing seven losses and a training strategy that alternates synthetic and real batches. Quantitative results show the RGB-only method achieves competitive 3D pose metrics compared to state-of-the-art 6D methods and uniquely provides the new DoFs for pedals and steering, along with 3D keypoints, though depth information would further improve translation accuracy. Overall, the work enables finer-grained pose and travel-direction estimation for articulated bicycles, with potential applications in cyclist intention prediction and collision avoidance in autonomous driving.

Abstract

In Autonomous Driving, cyclists belong to the safety-critical class of Vulnerable Road Users (VRU), and accurate estimation of their pose is critical for cyclist crossing intention classification, behavior prediction, and collision avoidance. Unlike rigid objects, articulated bicycles are composed of movable rigid parts linked by joints and constrained by a kinematic structure. 6D pose methods can estimate the 3D rotation and translation of rigid bicycles, but 6D becomes insufficient when the steering/pedals angles of the bicycle vary. That is because: 1) varying the articulated pose of the bicycle causes its 3D bounding box to vary as well, and 2) the 3D box orientation is not necessarily aligned to the orientation of the steering which determines the actual intended travel direction. In this work, we introduce a method for category-level 8D pose estimation for articulated bicycles and cyclists from a single RGB image. Besides being able to estimate the 3D translation and rotation of a bicycle from a single image, our method also estimates the rotations of its steering handles and pedals with respect to the bicycle body frame. These two new parameters enable the estimation of a more fine-grained bicycle pose state and travel direction. Our proposed model jointly estimates the 8D pose and the 3D Keypoints of articulated bicycles, and trains with a mix of synthetic and real image data to generalize on real images. We include an evaluation section where we evaluate the accuracy of our estimated 8D pose parameters, and our method shows promising results by achieving competitive scores when compared against state-of-the-art category-level 6D pose estimators that use rigid canonical object templates for matching.

Figures (4)

• Figure 1: Category-level 6D pose estimators can find an approximate match between an observed articulated posed cyclist (a) and a rigid canonical cyclist (b) template. But that introduces two problems: 1) A mis-match between the 3D bounding boxes of the canonical rigid template (c)-dashed and the articulated posed cyclist (c)-solid. 2) The cyclist travel directions (cyan arrows) are quite different from each other (c) v/s (b), although the dashed and solid boxes appear to be aligned. Ideally, an observed articulated posed cyclist (a) should be matched with an articulated posed template (c), as done in instance-level 6D methods.
• Figure 2: figure
• Figure 3: (a) Our camera setup. (b) Mean 3D Keypoints $\kappa_{\mu}$ (blue markers), and regressed 3D residual vectors $\Delta_{\kappa}$ (cyan) which point toward the red ground-truth 3D Keypoint $\bar{\kappa}_{3D}$ markers.
• Figure 4: Example qualitative results. (a) 2D Keypoints regressions from our $Dec_{2DK}$ decoder. Color codes in (a) are: blue: ground-truth, red: predictions. (b) Bicycle 8D poses estimated by our full model, including the regressed (lifted) 3D bicycle Keypoints (and their 3D-to-2D projections onto the image) for both synthetic and real unseen validation frames. Next to each image, we show three different camera views (Front, Top, Side) of the regressed (lifted) 3D Keypoints, in blue for better visualization. Typical failure cases are shown at the bottom row of (b).