Pulp Motion: Framing-aware multimodal camera and human motion generation

TL;DR

This paper addresses the challenge of jointly generating human motion and camera trajectories conditioned on text by introducing an auxiliary on-screen framing modality as a bridge between the two modalities. It presents a model-agnostic latent-diffusion framework with a joint autoencoder that maps motion and camera latents to a framing latent via a learnable matrix W, and introduces an auxiliary sampling mechanism that leverages this framing latent to steer generation toward coherent, on-screen framing. The PulpMotion dataset is introduced to support joint training and evaluation, featuring rich captions, longer sequences, and higher-quality motion. Across DiT- and MAR-based architectures, the method yields improved framing coherence and stronger text–modality alignment while maintaining strong per-modality generation, establishing state-of-the-art results for on-frame coherent joint motion–camera generation. These results have practical implications for cinematography-enabled AI systems by enabling more realistic and well-framed automated production pipelines.

Abstract

Treating human motion and camera trajectory generation separately overlooks a core principle of cinematography: the tight interplay between actor performance and camera work in the screen space. In this paper, we are the first to cast this task as a text-conditioned joint generation, aiming to maintain consistent on-screen framing while producing two heterogeneous, yet intrinsically linked, modalities: human motion and camera trajectories. We propose a simple, model-agnostic framework that enforces multimodal coherence via an auxiliary modality: the on-screen framing induced by projecting human joints onto the camera. This on-screen framing provides a natural and effective bridge between modalities, promoting consistency and leading to more precise joint distribution. We first design a joint autoencoder that learns a shared latent space, together with a lightweight linear transform from the human and camera latents to a framing latent. We then introduce auxiliary sampling, which exploits this linear transform to steer generation toward a coherent framing modality. To support this task, we also introduce the PulpMotion dataset, a human-motion and camera-trajectory dataset with rich captions, and high-quality human motions. Extensive experiments across DiT- and MAR-based architectures show the generality and effectiveness of our method in generating on-frame coherent human-camera motions, while also achieving gains on textual alignment for both modalities. Our qualitative results yield more cinematographically meaningful framings setting the new state of the art for this task. Code, models and data are available in our \href{https://www.lix.polytechnique.fr/vista/projects/2025_pulpmotion_courant/}{project page}.

Figures (26)

• Figure 1: Overview of our proposed auxiliary sampling. We adapt the joint generation of $({\mathbf{x}}, {\mathbf{y}})$ (camera trajectories and human motion) by leveraging an auxiliary modality ${\mathbf{z}}$ (on-screen human framing) to steer sampling toward more coherent joint generation via an orthogonal projection ${\bm{P}}_{/\!\!/}$. Specifically, our diffusion model predicts noise $\bm{\varepsilon}_\theta({\mathbf{x}},{\mathbf{y}})$, which is then adjusted along the auxiliary guidance direction.
• Figure 2: Architecture of the multimodal autoencoder. Human motion ${\mathbf{x}}_{\text{raw}}$ and camera trajectory ${\mathbf{y}}_{\text{raw}}$ are jointly encoded by $E_\phi$, linearly transformed via ${\bm{W}}$ into an auxiliary on-screen framing latent ${\mathbf{z}}$. Three decoders $D_{\psi_{\textnormal{x}}}$, $D_{\psi_{\textnormal{y}}}$, and $D_{\psi_{\textnormal{z}}}$ reconstruct raw modalities: $\hat{{\mathbf{x}}}_{\text{raw}}, \hat{{\mathbf{y}}}_{\text{raw}}, \hat{{\mathbf{z}}}_{\text{raw}}$.
• Figure 3: Decomposition of ${\mathbf{u}} = [{\mathbf{x}},{\mathbf{y}}]^\top$.${\mathbf{u}}$ decomposes onto two orthogonal components ${\mathbf{u}}_\perp$ and ${\mathbf{u}}_{/\!\!/}$. Our auxiliary sampling leverages this to encourage samples along ${\mathbf{u}}_{/\!\!/}$, parallel to the auxiliary modality ${\mathbf{z}}$.
• Figure 4: Comparison in DiT on the mixed set. Framing quality and modality-text alignment for ${\bm{c}}$ guidance ranges from 5 to 11. The optimal region is at the bottom-right (low framing FD, high alignment).
• Figure 5: Comparison in MAR on the mixed set. Framing quality and modality-text alignment for ${\bm{c}}$ guidance ranges from 1 to 5. The optimal region is at the bottom-right (low framing FD, high alignment).

Theorems & Definitions (3)

• Lemma 3.1
• Theorem C.1: Cochran
• proof