SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization

TL;DR

This work tackles the problem of aligning text-to-motion generation with human preferences by analyzing Direct Preference Optimization (DPO) in online and offline settings and identifying their complementary weaknesses. It introduces SoPo, a semi-online preference optimization framework that blends offline high-quality motions with online diverse unpreferred motions to overcome offline overfitting and online sampling bias. The method yields consistent improvements in alignment metrics (R-Precision, MM Dist) and generation quality across HumanML3D and KIT-ML, and extends to diffusion-based motion models with dedicated SoPo objectives. Empirical results, ablations, and reward-hacking analyses demonstrate that SoPo provides a practical, data-efficient path to more human-preferred text-to-motion generation, with potential impact on entertainment, virtual reality, and robotics applications.

Abstract

Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions, a critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models using "semi-online" data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other's limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25% (vs e.g. 0.76% of MoDiPO) on the MLD model, 2.91% (vs e.g. 0.66% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms of R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment. Project page: https://xiaofeng-tan.github.io/projects/SoPo/ .

Figures (7)

• Figure 1: Visual results on HumanML3D dataset. We integrate our SoPo into MDM Tevet2023 and MLD Chen2023, respectively. Our SoPo improves the alignment between text and motion preferences.
• Figure 2: Overfitting in offline DPO: green/red points are preferred/unpreferred motions; blue shows bias from fixed unpreferred data, red indicates uncovered unpreferred regions.
• Figure 3: Comparison of offline, online DPO, and our SoPo on synthetic data. Offline DPO suffers from mining unpreferred motions with high probability, and online DPO is limited by biased sampling. Our SoPo utilizes the dynamic unpreferred motions and preferred motions from unbiased offline dataset, overcoming their advantage. Here, the blue region is the distribution of generative model.
• Figure 4: Quantitative results on (a) spatial-preception motion generation, and (b) user study.
• Figure S1: Visual results on HumanML3D dataset. We integrate our SoPo into MDM Tevet2023 and MLD Chen2023, respectively. Our SoPo improves the alignment between text and motion preferences. Here, the red text denotes descriptions inconsistent with the generated motion.

Theorems & Definitions (11)

• Theorem 1
• Theorem 2
• proof
• proof
• proof
• proof
• proof
• Lemma 1
• Lemma 2
• Lemma 3