Rewriting Video: Text-Driven Reauthoring of Video Footage

TL;DR

This work proposes text-driven video reauthoring by reverse-engineering input video into an editable textual script and providing Rewrite Kit to manipulate this script. The generative reconstruction algorithm operates in an iterative loop (seed prompt generation, synthesis, comparison, refinement) and typically converges within 3–6 iterations, revealing a human–AI perceptual gap where temporal coherence matters more to humans than frame-level fidelity. A qualitative probe with 12 creators demonstrates novel use cases (virtual reshooting, editorial remixing, stylistic restyling, world-building) and highlights tensions around world coherence, authenticity, translation, and modality. The Rewrite Kit interface and study insights offer design implications for future co-creative tools that balance interpretability, control, and semantic generation, enabling more accessible and expressive forms of video storytelling while addressing ethical considerations of manipulation and provenance.

Abstract

Video is a powerful medium for communication and storytelling, yet reauthoring existing footage remains challenging. Even simple edits often demand expertise, time, and careful planning, constraining how creators envision and shape their narratives. Recent advances in generative AI suggest a new paradigm: what if editing a video were as straightforward as rewriting text? To investigate this, we present a tech probe and a study on text-driven video reauthoring. Our approach involves two technical contributions: (1) a generative reconstruction algorithm that reverse-engineers video into an editable text prompt, and (2) an interactive probe, Rewrite Kit, that allows creators to manipulate these prompts. A technical evaluation of the algorithm reveals a critical human-AI perceptual gap. A probe study with 12 creators surfaced novel use cases such as virtual reshooting, synthetic continuity, and aesthetic restyling. It also highlighted key tensions around coherence, control, and creative alignment in this new paradigm. Our work contributes empirical insights into the opportunities and challenges of text-driven video reauthoring, offering design implications for future co-creative video tools.

Figures (8)

• Figure 1: The average similarity across iterations and the rationale for early stopping. The solid line shows the mean similarity score if we keep iterating uniformly, while the dashed line shows the "best-so-far" average, calculated as if each clip stops the first time it reaches its peak score. Performance gains typically saturate by 3-6 iterations. Further iterations often lower the per-iteration average due to "prompt drift," whereas the best-so-far curve remains flat, demonstrating the benefit of implementing per-clip early stopping.
• Figure 2: The user interface of the Rewrite Kit technology probe. A creator's three-part workflow: reverse-engineer, rewrite, and generate.
• Figure 3: Use case 12: Camera angle change. Change the fairy shot to a low angle from the ground.
• Figure 4: Use case 14: Generate a smooth transition between two clips. Note that the middle two rows of transition are all 100% generated.
• Figure 5: Use case 9: Stylizing a vlog as pixel art. Guided by a single reference image, Rewrite Kit restyles the original photorealistic footage into a pixel art animation.