Performance-Based Biped Control using a Consumer Depth Camera

TL;DR

The paper tackles real-time control of a physically simulated biped using free-form, noisy depth-camera input. It introduces a parameterized controller that upper-body poses are copied while lower-body balance is maintained via a pelvis-height-based IK linked to a base walking motion, with a 3D parameter space $\mathbf P=(P_x, P_y, P_z)$ and per-reference-motion controller gains $\boldsymbol\theta^i$ optimized offline by CMA. Real-time operation selects $\boldsymbol\theta^{\mathrm{cur}}$ through regression over precomputed points using inverse-distance weighting, enabling the biped to imitate user poses while walking stably. Comparisons show superior robustness of the parameterized controller over a baseline, and demonstrations highlight practical interactive capabilities for immersive VR/game applications, albeit with offline computational cost and sensitivity to input quality.

Abstract

We present a technique for controlling physically simulated characters using user inputs from an off-the-shelf depth camera. Our controller takes a real-time stream of user poses as input, and simulates a stream of target poses of a biped based on it. The simulated biped mimics the user's actions while moving forward at a modest speed and maintaining balance. The controller is parameterized over a set of modulated reference motions that aims to cover the range of possible user actions. For real-time simulation, the best set of control parameters for the current input pose is chosen from the parameterized sets of pre-computed control parameters via a regression method. By applying the chosen parameters at each moment, the simulated biped can imitate a range of user actions while walking in various interactive scenarios.

Figures (10)

• Figure 1: The depth camera in front of the user captures the user's pose and our system controls a simulated biped as shown on the monitor.
• Figure 2: The user pose (left) is converted to the target pose (right). Left: A user is bending his knees while moving his arms. Center: The difference in the pelvis height (d) of the user pose (red) and the upright-standing pose (green) is applied to the lower-body part of the target pose. Right: The resulting target pose makes the simulated biped mimic the user's upper-body pose and knee-bending while walking.
• Figure 3: The parameters for controller parameterization. Left: $\mathbf P_x$ and $\mathbf P_z$ are the horizontal offset of the upper-body CM of the input pose (red cube) from that of the current pose of the base motion (blue cube). Right: $\mathbf P_y$ is the offset of the pelvis height of the input pose (red) from that of the standing pose (green).
• Figure 4: Parameterization space. Each point $\mathbf P^i$ in the space indicates a reference motion used in the optimization process. The modulated reference poses (1), (2), (3), (4), (5), and (6) correspond to the first frames of the reference motions indicated by the sample parameters $\mathbf P^1=(0,0,0)$, $\mathbf P^2=(0.1,0,-0.2)$, $\mathbf P^3=(0.2,-0.1,0)$, $\mathbf P^4=(0,-0.2,0)$, $\mathbf P^5=(0.1,-0.1,0.1)$, and $\mathbf P^6=(0.2,-0.2,0)$, respectively. The points are rendered using three different colors based on their $x$ values for a clean visualization.
• Figure 5: Optimization of parameter space. The range of the initial parameter space is rendered as the green cube (a). Minimum evaluated values of the first 45 sample points (b), of 21 sample points scaled by $0.8$ (c), of 5 sample points scaled by $0.8^2$ (d), of 1 sample point scaled by $0.8^4$ (e). Successful points (evaluated value from $2000$ to $3000$) are rendered in color from red (value of $2000$ or less) to purple (up to value of $3000$) and failed points (evaluated value more than $3000$) are rendered in color from blue (value of $3000$) to black (value of $10000$ or more).