CAMO: Category-Agnostic 3D Motion Transfer from Monocular 2D Videos

TL;DR

CAMO addresses 2D-to-3D motion transfer across diverse categories by bypassing intermediate 3D reconstructions and category priors. It introduces a morphology-adaptive shape parameterization coupled with Articulated Gaussian Splatting, optimized directly in image space with dense semantic correspondences via differentiable rendering. By learning bone lengths $\ell_b$, global scale $s_{global}$, and local offsets $\boldsymbol{o}_i$ alongside a time-conditioned pose model, CAMO disentangles shape from pose and preserves topology. Empirically, it achieves superior PMD and FID across Mixamo and DT4D and works robustly on real monocular videos, demonstrating strong generalization to novel morphologies and casual capture conditions.

Abstract

Motion transfer from 2D videos to 3D assets is a challenging problem, due to inherent pose ambiguities and diverse object shapes, often requiring category-specific parametric templates. We propose CAMO, a category-agnostic framework that transfers motion to diverse target meshes directly from monocular 2D videos without relying on predefined templates or explicit 3D supervision. The core of CAMO is a morphology-parameterized articulated 3D Gaussian splatting model combined with dense semantic correspondences to jointly adapt shape and pose through optimization. This approach effectively alleviates shape-pose ambiguities, enabling visually faithful motion transfer for diverse categories. Experimental results demonstrate superior motion accuracy, efficiency, and visual coherence compared to existing methods, significantly advancing motion transfer in varied object categories and casual video scenarios.

Figures (8)

• Figure 1: Conceptual overview of CAMO. Our method directly transfers articulated motion from 2D video to diverse target objects, without requiring 3D reconstruction of the source or any parametric templates.
• Figure 2: Overview of the morphology-adaptive articulated Gaussian splatting pipeline. Given a target mesh, we parameterize it with deformable 3D Gaussians. A time-conditioned MLP ($f_{\text{MLP}}$) predicts skeletal transformations driven by input time embeddings. Crucially, our pipeline employs morphology adaptation (Sec. \ref{['sec:3.2']}) to align the target's canonical structure, followed by LBS-based deformation (Sec. \ref{['sec:3.1']}) for articulation. The framework is optimized end-to-end using differentiable rendering ($\mathcal{L}_{\text{render}}$) and semantic keypoint constraints ($\mathcal{L}_{\text{keypoint}}$) consistent with the source video.
• Figure 3: Deformable morphology parameterization.(a) We initialize the target character with skeleton rigging, acquiring the topological structure and skinning weights. (b) Morphology-adaptive parameterization of structural variations. (c) During optimization, shape parameters deform the target's morphological structure to align with the morphology of the source.
• Figure 4: Dense target-source correspondences matching. We extract robust 2D-to-3D semantic correspondences by matching semantic features between source frames and rendered target views.
• Figure 5: Qualitative results on Mixamo and DT4D-Quadruped datasets. Our method shows superior pose alignment compared to baselines across diverse objects. Refer to the supplementary video for full animation.