Volumetric Temporal Texture Synthesis for Smoke Stylization using Neural Cellular Automata

TL;DR

This work tackles the challenge of stylizing 3D volumetric smoke with spatiotemporal coherence and efficiency. It introduces Volumetric Neural Cellular Automata (VNCA), a lightweight, trainable 3D texture generator that is trained on a single density frame yet can stylize the entire sequence in real time by leveraging self‑emerging motion and an Eulerian rendering pipeline. VNCA combines a perception‑based update rule with density and velocity priors and a differentiable volume renderer, supervised by a deep style loss and flow‑guided motion terms, achieving real‑time, multi‑view coherent stylization and generalization to unseen data. The approach speeds up training by over an order of magnitude compared to prior volumetric NST methods and can be extended to mesh texturing, offering a practical tool for artists and graphics pipelines. The results demonstrate strong appearance match to reference styles, temporally smooth transitions, and plausible motion consistent with the input smoke dynamics.

Abstract

Artistic stylization of 3D volumetric smoke data is still a challenge in computer graphics due to the difficulty of ensuring spatiotemporal consistency given a reference style image, and that within reasonable time and computational resources. In this work, we introduce Volumetric Neural Cellular Automata (VNCA), a novel model for efficient volumetric style transfer that synthesizes, in real-time, multi-view consistent stylizing features on the target smoke with temporally coherent transitions between stylized simulation frames. VNCA synthesizes a 3D texture volume with color and density stylization and dynamically aligns this volume with the intricate motion patterns of the smoke simulation under the Eulerian framework. Our approach replaces the explicit fluid advection modeling and the inter-frame smoothing terms with the self-emerging motion of the underlying cellular automaton, thus reducing the training time by over an order of magnitude. Beyond smoke simulations, we demonstrate the versatility of our approach by showcasing its applicability to mesh stylization.

Figures (12)

• Figure 1: Given a simulated 3D smoke sequence and a reference style image, we train our Volumetric Neural Cellular Automata (VNCA) model on a single density frame of the simulation to synthesize a dynamic texture volume. For inference, our trained model can stylize the whole smoke sequence in real-time. We recommend the readers check our images on screen for optimized visual quality.
• Figure 2: Update Rule. We use the update rule to determine the cell state at the next time step. We obtain the perception vector with neighboring cell state and concatenate density and positional encoding as priors. The stochastic update processes this vector for an update to each cell.
• Figure 3: Training. At training, we use a single input frame for the Density Encoding shown in Fig. 2. At each epoch, we apply the VNCA update rule for $n$ steps and record the cell state before and after the update. We then render the smoke stylized with cell state, using Eq.7, before and after the VNCA updates to get $P_b$ and $P_f$. We use $P_f$ to match the style image and extract motion between $P_b$ and $P_f$ to align with the fluid motion.
• Figure 4: Qualitative Stylization Comparison with TNST and LNST. VNCA synthesizes stylization that closely matches the reference image, with superior quality than TNST and LNST indicated by our user study.
• Figure 5: Between-frame Coherence Comparison with TNST and LNST. VNCA synthesizes more coherent stylization across frames, compared to per-frame optimization methods.