SuDA: Support-based Domain Adaptation for Sim2Real Motion Capture with Flexible Sensors

TL;DR

The paper addresses the high cost and scarcity of labeled real data for flexible-sensor MoCap by introducing SuDA, a support-based domain adaptation method that aligns predictive-function supports, not data distributions, to enable a Sim2Real transfer without real labels. SuDA uses a Body-Fabric-Sensor simulation to generate source data and pairs it with unlabeled real sensor data, employing a support-registration mechanism $R^Q$ that maps simulated sensor supports to real supports via $n+1$ proxy points. The authors demonstrate that SuDA achieves comparable accuracy to supervised learning and substantially outperforms state-of-the-art distribution-based DA methods across diverse users, motions, wearing positions, and real-world activities, all while using no real labeled data. The approach shows strong robustness and practical potential for low-dimensional MoCap tasks and points to broader applicability to other low-dimensional domains where collecting labeled data is expensive.

Abstract

Flexible sensors hold promise for human motion capture (MoCap), offering advantages such as wearability, privacy preservation, and minimal constraints on natural movement. However, existing flexible sensor-based MoCap methods rely on deep learning and necessitate large and diverse labeled datasets for training. These data typically need to be collected in MoCap studios with specialized equipment and substantial manual labor, making them difficult and expensive to obtain at scale. Thanks to the high-linearity of flexible sensors, we address this challenge by proposing a novel Sim2Real Mocap solution based on domain adaptation, eliminating the need for labeled data yet achieving comparable accuracy to supervised learning. Our solution relies on a novel Support-based Domain Adaptation method, namely SuDA, which aligns the supports of the predictive functions rather than the instance-dependent distributions between the source and target domains. Extensive experimental results demonstrate the effectiveness of our method andits superiority over state-of-the-art distribution-based domain adaptation methods in our task.

Figures (14)

• Figure 1: (a) Our Sim2Real approach simulates flexible sensors to predict joint angle in the real world. (b) Distribution Registration registers the feature distributions $Dist(P(x_s))$ and $Dist(P(x_t))$ of the source and target domains, which fails to match points of the same label $y_s$$=$$y_t$ when the two distributions are inherently different. In contrast, the proposed Support Registration registers the function supports $supp(f_s)$ and $supp(f_t)$ that are independent of the specific distributions, thereby guaranteeing successful domain adaptation.
• Figure 2: (a) Our hardware. $R_1$ and $R_2$ denote the two sensors on our smart pad, and $\theta$ denotes the joint angle. (b) High linearity characteristics of capacitive strain sensors.
• Figure 3: Pipeline Overview. Top row: the generation of simulation data with our Body-Fabric-Sensor model, which consists of four main parts: 1. Body Modeling 2. Fabric Deformation 3. Elbow Bending Angle 4. Sensor Streching. Bottom row: the proposed support-based domain adaptation method (SuDA). SuDA first parameterizes the supports of source and target domain, denoted as $supp(f_s)$ and $supp(f_t)$, into $n+1$ evenly-distributed proxy points respectively, denoted as $g_s^Q(l)$ and $g_t^Q(l)$. Then, it applies a novel Support Registration $R^Q$ to map $x_s$ to its nearest $p_s^i$. Finally, we train the predictive function $f^*_t$ on the target domain using ($\hat{x}_t$, $\hat{y}_t=$$y_s$) and finally to $\hat{x}_t$.
• Figure 4: Orange and Blue: source and target data samples. Black: parameterized and quantized supports. (a) Two scatter curve tracks are our data samples and represent function support of the two domains. (b) Then, we use curve parameterization to quantize function support into several segments. Since $\frac{df_s}{dx_s} \approx \frac{df_t}{dx_t}$, points with the same parameters have the same labels and can be registered together by $R^Q$.
• Figure 5: MAE vs. size of simulation dataset. MAE drops quickly between 0 and 5,000 frames with the increase in the size of the simulation dataset. Between 5,000 and 10,000 frames, the MAE drops slightly towards convergence. With more than 10,000 frames, MAE becomes stable against the increase in dataset size.

Theorems & Definitions (1)

• Definition 4.1: SuDA