Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation

Mert Kiray, Paul Uhlenbruck, Nassir Navab, Benjamin Busam

TL;DR

PromptVFX reframes 3D animation as applying time-varying $4$-D fields to Gaussian splats, enabling real-time, text-driven edits without diffusion or physics simulations. An LLM translates natural language prompts into parametric functions that govern Gaussian centers, colors, and opacities, while a LVLM grounding step aligns appearance with the scene; multiple hypotheses and iterative refinement via VLMs and user feedback ensure alignment with user intent. The approach delivers fast, interactive 3D effects on consumer hardware and in browser environments, demonstrated against diffusion- and physics-based baselines with favorable perceptual and temporal coherence metrics. This work advances democratized 3D content creation by providing a scalable, training-free pipeline that couples language interfaces with efficient, parameter-driven 3D representations, though it acknowledges ethical considerations around synthetic media and outlines avenues for future physics-informed enhancements.

Abstract

Visual effects (VFX) are key to immersion in modern films, games, and AR/VR. Creating 3D effects requires specialized expertise and training in 3D animation software and can be time consuming. Generative solutions typically rely on computationally intense methods such as diffusion models which can be slow at 4D inference. We reformulate 3D animation as a field prediction task and introduce a text-driven framework that infers a time-varying 4D flow field acting on 3D Gaussians. By leveraging large language models (LLMs) and vision-language models (VLMs) for function generation, our approach interprets arbitrary prompts (e.g., "make the vase glow orange, then explode") and instantly updates color, opacity, and positions of 3D Gaussians in real time. This design avoids overheads such as mesh extraction, manual or physics-based simulations and allows both novice and expert users to animate volumetric scenes with minimal effort on a consumer device even in a web browser. Experimental results show that simple textual instructions suffice to generate compelling time-varying VFX, reducing the manual effort typically required for rigging or advanced modeling. We thus present a fast and accessible pathway to language-driven 3D content creation that can pave the way to democratize VFX further. Code available at https://obsphera.github.io/promptvfx/.

PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation

TL;DR

PromptVFX reframes 3D animation as applying time-varying -D fields to Gaussian splats, enabling real-time, text-driven edits without diffusion or physics simulations. An LLM translates natural language prompts into parametric functions that govern Gaussian centers, colors, and opacities, while a LVLM grounding step aligns appearance with the scene; multiple hypotheses and iterative refinement via VLMs and user feedback ensure alignment with user intent. The approach delivers fast, interactive 3D effects on consumer hardware and in browser environments, demonstrated against diffusion- and physics-based baselines with favorable perceptual and temporal coherence metrics. This work advances democratized 3D content creation by providing a scalable, training-free pipeline that couples language interfaces with efficient, parameter-driven 3D representations, though it acknowledges ethical considerations around synthetic media and outlines avenues for future physics-informed enhancements.

Abstract

Visual effects (VFX) are key to immersion in modern films, games, and AR/VR. Creating 3D effects requires specialized expertise and training in 3D animation software and can be time consuming. Generative solutions typically rely on computationally intense methods such as diffusion models which can be slow at 4D inference. We reformulate 3D animation as a field prediction task and introduce a text-driven framework that infers a time-varying 4D flow field acting on 3D Gaussians. By leveraging large language models (LLMs) and vision-language models (VLMs) for function generation, our approach interprets arbitrary prompts (e.g., "make the vase glow orange, then explode") and instantly updates color, opacity, and positions of 3D Gaussians in real time. This design avoids overheads such as mesh extraction, manual or physics-based simulations and allows both novice and expert users to animate volumetric scenes with minimal effort on a consumer device even in a web browser. Experimental results show that simple textual instructions suffice to generate compelling time-varying VFX, reducing the manual effort typically required for rigging or advanced modeling. We thus present a fast and accessible pathway to language-driven 3D content creation that can pave the way to democratize VFX further. Code available at https://obsphera.github.io/promptvfx/.

Paper Structure

This paper contains 27 sections, 2 equations, 24 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. 3D Representation via Gaussian Splatting
  5. Time-Dependent Fields for Animation
  6. LLM-Based Translation of Text Instructions
  7. Design Phase
  8. Field Generation Phase
  9. Animation Hypotheses Generation & Scoring
  10. Automatic Refinement
  11. Conversational Refinement
  12. Experiments
  13. Experimental Details
  14. Qualitative Evaluation
  15. Quantitative Evaluation
  16. ...and 12 more sections

Figures (24)

  • Figure 1: Overview of the PromptVFX pipeline. Given a user-provided textual prompt, the system first decomposes it into structured animation phases. A large language model (LLM) then generates parametric functions that define the motion, color, and opacity changes of 3D Gaussians over time. To handle ambiguity, multiple animation hypotheses are generated and evaluated using a vision-language model (VLM) or user feedback. The selected animation is further refined through automatic and interactive text-based corrections, ensuring high-quality, real-time results. More details on prompt formulation and scoring are provided in the supplementary material.
  • Figure 2: Qualitative results showcasing the diversity and fidelity of animations generated by PromptVFX across different scenes and user prompts. Exact prompts used to generate these animations are provided in the supplementary material.
  • Figure 3: Qualitative comparison with baselines on different scenes and user prompts. Our method achieves high-fidelity visual transformations and realistic motion, outperforming AutoVFX and Gaussians2Life. Exact prompts and additional qualitative comparisons are provided in the supplementary material.
  • Figure 4: Impact of VLM feedback on animation accuracy. Without visual feedback, the bulldozer incorrectly accelerates off the table, failing to account for scene constraints. With VLM refinement, the motion is corrected to remain contextually appropriate.
  • Figure 5: Animation refinement is demonstrated through iterative user interaction. After generating an initial animation, the system enables users to provide follow-up prompts for further adjustments, enhancing control and precision in the final animation.
  • ...and 19 more figures