EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation

TL;DR

The paper tackles stereo egocentric 3D pose estimation under joint invisibility caused by limited head-mounted camera FOV. It introduces EgoPoseFormer, a simple two-stage architecture with a Pose Proposal Network for coarse joint localization and a Pose Refinement Transformer that uses Joint Query Tokens and Deformable Stereo Attention to refine poses via multi-view stereo cues. The approach achieves state-of-the-art results on UnrealEgo and strong monocular performance on SceneEgo, while remaining computationally efficient. Notably, the PPN alone can outperform prior methods with proper training, underscoring the importance of training strategy and the effectiveness of a transformer-based refinement stage.

Abstract

We present EgoPoseFormer, a simple yet effective transformer-based model for stereo egocentric human pose estimation. The main challenge in egocentric pose estimation is overcoming joint invisibility, which is caused by self-occlusion or a limited field of view (FOV) of head-mounted cameras. Our approach overcomes this challenge by incorporating a two-stage pose estimation paradigm: in the first stage, our model leverages the global information to estimate each joint's coarse location, then in the second stage, it employs a DETR style transformer to refine the coarse locations by exploiting fine-grained stereo visual features. In addition, we present a Deformable Stereo Attention operation to enable our transformer to effectively process multi-view features, which enables it to accurately localize each joint in the 3D world. We evaluate our method on the stereo UnrealEgo dataset and show it significantly outperforms previous approaches while being computationally efficient: it improves MPJPE by 27.4mm (45% improvement) with only 7.9% model parameters and 13.1% FLOPs compared to the state-of-the-art. Surprisingly, with proper training settings, we find that even our first-stage pose proposal network can achieve superior performance compared to previous arts. We also show that our method can be seamlessly extended to monocular settings, which achieves state-of-the-art performance on the SceneEgo dataset, improving MPJPE by 25.5mm (21% improvement) compared to the best existing method with only 60.7% model parameters and 36.4% FLOPs. Code is available at: https://github.com/ChenhongyiYang/egoposeformer .

Figures (9)

• Figure 1: Illustration of different egocentric pose estimation methods. While previous approaches predict joints' locations via 2D heatmaps or 3D feature voxels, EgoPoseFormer first estimates the coarse locations of each joint using a Pose Proposal Network (PPN) and uses a transformer to refine the estimated pose.
• Figure 2: (a) An overview of the proposed EgoPoseFormer. The input of EgoPoseFormer is the multi-view image features. In the first stage: Pose Proposal Network (PPN), the multi-view features are globally pooled and concatenated, from which an MLP is used to estimate the coarse location of each joint (pose proposal). Then the joints' identity and location information are embedded into Joint Query Tokens (JQTs) to serve as the queries in the second stage Pose Refinement Transformer (PRFormer). In PRFormer, a JQT iteratively interacts with the stereo features and other JQTs to update itself through the attention mechanism. The updated JQTs are used to predict refinement offsets related to the pose proposal, yielding more accurate pose estimations. (b) The architecture of PRFormer layer is similar to the transformer decoder layer, which includes a cross-attention block, a self-attention block, and a feed-forward network (FFN). However, in PRFormer, the cross-attention is replaced by the proposed Deformable Stereo Attention to better exploit stereo visual features.
• Figure 3: Left: An illustration of our Deformable Stereo Attention. The 3D joints are first projected to each view plane using camera parameters. Within each view, we compute 2D deformable attention by querying the image features with the JQTs with the projected points serving as reference points. Finally, the attention results for each view are concatenated and fed into a linear layer to be projected into the original dimension. Right: (a) The feature extractor is first pre-trained to predict 2D joint heatmaps using monocular images. (b) The multi-view feature maps are computed using the pre-trained feature extractor.
• Figure 4: Qualitative visualization on UnrealEgo and SceneEgo. Ground-truths are colored in green and predictions are colored in red.
• Figure 5: Error analysis of the pose proposal and the refined body pose estimation.