BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds

TL;DR

BGM2Pose tackles the challenge of non-invasively estimating 3D human poses from non-stationary everyday music by introducing a dual strategy: a Frequency-wise Attention mechanism that focuses on informative frequency bands and a Contrastive Pose Extraction module that separates pose signals from music via hard negative sampling. The approach is instantiated on the AMPL dataset, demonstrating improved RMSE, MAE, and PCKh@0.5 over baselines and showing robustness to unseen music and plain clothes. This work advances practical acoustic sensing by enabling pose estimation in naturalistic environments without intrusive signals, with potential applications in privacy-preserving surveillance, VR/AR, and healthcare. The combination of FA and CPE provides a general template for extracting task-relevant information from complex, non-stationary audio sources.

Abstract

We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes natural music that causes minimal discomfort to humans. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Our datasets and code will be made publicly available.

Figures (7)

• Figure 1: We propose BGM2Pose, a 3D human pose estimation that utilizes background music that naturally integrates into everyday life, for active acoustic sensing. Given playback music and recorded acoustic signals, our method estimates frame-by-frame 3D human poses. Unlike chirp signals, BGM tracks contain unstable sounds and limited frequency bands.
• Figure 2: Acoustic feature visualization. (Left) With the repeated chirp signals, differences in collected audio data correspond directly to pose changes, forming patterns for each pose. (Right) In our setting, changes in BGM overshadow human pose variation.
• Figure 3: The overview of our framework for BGM–based 3D human pose estimation.
• Figure 4: We perform contrastive learning using encoded pose sequences and recorded audio features to achieve feature extraction that is independent of the BGM.
• Figure 5: The setup of our experiments.