Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffusionCinema: Text-to-Aerial Cinematography

Valerii Serpiva, Artem Lykov, Jeffrin Sam, Aleksey Fedoseev, Dzmitry Tsetserukou

TL;DR

The paper presents DiffusionCinema, a framework that converts natural language prompts and an initial image into autonomous, cinematic UAV trajectories by leveraging diffusion-based video priors and a Vision-Language Navigation (VLN) backbone. A Diffusion Transformer with a multimodal Cosmos-Reason1 module conditions video synthesis on text and imagery to produce a syn video and associated latent representations, which are aligned with real UAV dynamics through visual odometry and a PID control loop. A preliminary user study (N=10) shows substantially lower workload and frustration for the text-driven interface compared with manual remote control, supporting the approach's usability and practicality for rapid, cinematic aerial content creation. The work introduces a practical, scalable paradigm—text-driven aerial cinematography—that couples cinematic priors from human-shot video with autonomous flight, enabling non-experts to capture expressive drone footage efficiently.

Abstract

We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g., "orbit around me slowly from the right and reveal the background waterfall"). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface (M = 21.6) compared to a traditional remote controller (M = 58.1), demonstrating a substantial reduction in perceived effort. Mental demand (M = 11.5 vs. 60.5) and frustration (M = 14.0 vs. 54.5) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the "creative operator" converting story intentions directly into aerial motion.

DiffusionCinema: Text-to-Aerial Cinematography

TL;DR

The paper presents DiffusionCinema, a framework that converts natural language prompts and an initial image into autonomous, cinematic UAV trajectories by leveraging diffusion-based video priors and a Vision-Language Navigation (VLN) backbone. A Diffusion Transformer with a multimodal Cosmos-Reason1 module conditions video synthesis on text and imagery to produce a syn video and associated latent representations, which are aligned with real UAV dynamics through visual odometry and a PID control loop. A preliminary user study (N=10) shows substantially lower workload and frustration for the text-driven interface compared with manual remote control, supporting the approach's usability and practicality for rapid, cinematic aerial content creation. The work introduces a practical, scalable paradigm—text-driven aerial cinematography—that couples cinematic priors from human-shot video with autonomous flight, enabling non-experts to capture expressive drone footage efficiently.

Abstract

We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g., "orbit around me slowly from the right and reveal the background waterfall"). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface (M = 21.6) compared to a traditional remote controller (M = 58.1), demonstrating a substantial reduction in perceived effort. Mental demand (M = 11.5 vs. 60.5) and frustration (M = 14.0 vs. 54.5) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the "creative operator" converting story intentions directly into aerial motion.
Paper Structure (13 sections, 4 equations, 3 figures)

This paper contains 13 sections, 4 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-Driven Aerial Autonomy
  4. Vision-Language-Action Models
  5. Aerial Vision-Language Navigation
  6. Video Diffusion Models as Motion Priors
  7. System Overview
  8. Diffusion Model
  9. Path Extraction and UAV Flight Execution
  10. Preliminary User Evaluation
  11. User Study Setup
  12. User Study Results
  13. Conclusion and Future Work

Figures (3)

  • Figure 1: Generated video sequence for the task “Make a cinematic shot for my video vlog with me as the host.” The top row of images shows frames generated by the DiffusionCinema model, while the bottom row shows the corresponding real footage captured by the UAV following the host.
  • Figure 2: (a) NASA-TLX questionnaire. Black error bars denote 95% confidence interval (CI). (b) The UEQ questionnaire. Black error bars denote 95% CI. Values between -0.8 and 0.8 (in the dashed line) represent a neutral evaluation.
  • Figure 3: Comparison of the trajectory generated by ORB-SLAM from synthetic video and the trajectory executed by a real UAV: yellow line is the generated trajectory; blue line is the actual UAV path; gray circle-target object.