COMPOSE: Hypergraph Cover Optimization for Multi-view 3D Human Pose Estimation

TL;DR

COMPOSE tackles 3D multi-view human pose estimation with sparse camera setups by moving beyond pairwise associations to a higher-order hypergraph formulation. It models detections across multiple views as hyperedges and casts correspondence as a weighted exact-cover problem, solved efficiently via ILP with a geometry-driven pruning strategy. The approach achieves state-of-the-art results among optimization-based methods and competitive performance relative to self-supervised learning on standard benchmarks, demonstrating strong robustness to occlusions and detection noise. This higher-order, geometry-guided framework offers practical benefits for accurate 3D pose reconstruction without requiring extensive 3D supervision, with potential downstream impact on tracking and activity understanding in multi-camera setups.

Abstract

3D pose estimation from sparse multi-views is a critical task for numerous applications, including action recognition, sports analysis, and human-robot interaction. Optimization-based methods typically follow a two-stage pipeline, first detecting 2D keypoints in each view and then associating these detections across views to triangulate the 3D pose. Existing methods rely on mere pairwise associations to model this correspondence problem, treating global consistency between views (i.e., cycle consistency) as a soft constraint. Yet, reconciling these constraints for multiple views becomes brittle when spurious associations propagate errors. We thus propose COMPOSE, a novel framework that formulates multi-view pose correspondence matching as a hypergraph partitioning problem rather than through pairwise association. While the complexity of the resulting integer linear program grows exponentially in theory, we introduce an efficient geometric pruning strategy to substantially reduce the search space. COMPOSE achieves improvements of up to 23% in average precision over previous optimization-based methods and up to 11% over self-supervised end-to-end learned methods, offering a promising solution to a widely studied problem.

Figures (4)

• Figure 1: Top: Traditional approaches rely on pairwise geometric constraints dong_FastRobustMultiPerson_2022zhang_4DAssociationGraph_2020wu_GraphBased3DMultiPerson_2021. As illustrated, these methods generate pairwise associations that, while locally consistent between two views, often fail to form a globally coherent structure. Consequently, algorithms face the difficult task of reconciling these locally plausible but globally conflicting edges to recover the correct 3D poses. Bottom: We propose a hypergraph formulation that jointly models higher-order relationships across views. We re-frame correspondence matching as a hypergraph partitioning problem, where hyperedges encode multi-view consistency. This global formulation effectively resolves ambiguities by enforcing consensus across the entire set of views.
• Figure 2: An overview of COMPOSE. Multi-view images are taken as input (I). First, we employ an off-the-shelf 2D pose estimator to extract 2D keypoints (II). Next, we construct our weighted hypergraph (III). We then solve the weighted exact cover problem to optimally partition the graph and establish unique correspondences (IV). Finally, we triangulate the 3D keypoints from these correspondences to obtain our final 3D human poses (V).
• Figure 3: Qualitative results on the CMU panoptic dataset panoptic. We show input 2D detections obtained from ViTPose xu2022vitpose and the 3D predictions from MvPose dong_FastRobustMultiPerson_2022 and our method. 3D ground truths are visualized in red, predictions are visualized in other colors. As highlighted, MvPose fails to predict the correspondences for one person, while our method successfully reconstructs all individuals.
• Figure 4: Qualitative results on the Shelf dataset belagiannis_3DPictorialStructures_2014 (frame 305 and 544). We visualize the ground truth (dashed red lines) and our predictions (solid colored lines). Note that for the highlighted actors, the ground truth annotations deviate significantly from the image evidence, while our method recovers the visually correct poses. Best viewed in color and zoomed in.

Theorems & Definitions (2)

• Proposition 1
• proof