LILAC: Long-sequence Incremental Low-latency Arbitrary Motion Stylization via Streaming VAE-Diffusion with Causal Decoding

TL;DR

The paper addresses real-time, long-sequence motion stylization, a setting where offline diffusion-based methods struggle due to temporal inconsistency and lack of streaming capability. It introduces LILAC, a streaming latent-space VAE–Diffusion framework that uses a sliding-window architecture, latent re-decoding/encoding, and a causal decoder to achieve smooth, style-controlled motion without access to future frames. Key contributions include a formal problem definition preserving input trajectories, a practical streaming pipeline with a latent buffer and dynamic style conditioning, and comprehensive experiments demonstrating real-time performance and long-sequence temporal coherence, aided by targeted ablations. The work enables real-time, arbitrarily stylized motions for VR and animation, and its modular design promises compatibility with other VAE–Diffusion architectures and potential runtime gains from faster sampling techniques.

Abstract

Generating long and stylized human motions in real time is critical for applications that demand continuous and responsive character control. Despite its importance, existing streaming approaches often operate directly in the raw motion space, leading to substantial computational overhead and making it difficult to maintain temporal stability. In contrast, latent-space VAE-Diffusion-based frameworks alleviate these issues and achieve high-quality stylization, but they are generally confined to offline processing. To bridge this gap, LILAC (Long-sequence Incremental Low-latency Arbitrary Motion Stylization via Streaming VAE-Diffusion with Causal Decoding) builds upon a recent high-performing offline framework for arbitrary motion stylization and extends it to an online setting through a latent-space streaming architecture with a sliding-window causal design and the injection of decoded motion features to ensure smooth motion transitions. This architecture enables long-sequence real-time arbitrary stylization without relying on future frames or modifying the diffusion model architecture, achieving a favorable balance between stylization quality and responsiveness as demonstrated by experiments on benchmark datasets. Supplementary video and examples are available at the project page: https://pren1.github.io/lilac/

Figures (6)

• Figure 2: Proposed streaming stylization pipeline. Motion segments are processed by an encoder–decoder with a style-conditioned denoiser. The latent output of the diffusion model is re-decoded into motion features, which are then concatenated with previous outputs, corrected through trajectory copy, re-encoded, and finally passed into a causal decoder for temporally consistent motion generation.
• Figure 3: Stylization example. The generated sequence (right) dynamically switches between multiple styles (old, open-leg, and hands-high) in real time, while the ground-truth motion (left) — corresponding to the input reference — is shown for comparison. The color intensity indicates temporal progression, with darker poses corresponding to more recent frames. The small insets display the corresponding root trajectories, with a red dot indicating the end position of each trajectory.
• Figure 4: Qualitative comparison of long-sequence motion generation under the hands-high stylization condition. The ground-truth motion (left) shows natural continuity. The proposed streaming model (middle) produces smooth and temporally consistent stylized motion without jitter, while the offline baseline (right) suffers from temporal jitter due to independent generation. Visualization conventions (e.g., color intensity and trajectory insets) follow those in Fig. \ref{['fig:stylization']}.
• Figure : (a) Baseline VAE
• Figure : (a) Baseline VAE