CineVerse: Consistent Keyframe Synthesis for Cinematic Scene Composition

TL;DR

CineVerse tackles cinematic scene composition by generating coherent multi-shot keyframe sequences with consistent characters and settings. It introduces a two-stage pipeline where an LLM performs in-context scene planning to produce a detailed shot-by-shot plan, followed by a fine-tuned text-to-image model to synthesize the keyframes. A CineVerse dataset is built by adapting Storyboard20K and enriching it with MovieNet context and LLM/MLLM-derived attributes to enable controlled generation. Empirical results show improved text-image alignment, inter-frame consistency, and frame-accurate shot counts compared with baselines, highlighting the potential of automated cinematic scene synthesis.

Abstract

We present CineVerse, a novel framework for the task of cinematic scene composition. Similar to traditional multi-shot generation, our task emphasizes the need for consistency and continuity across frames. However, our task also focuses on addressing challenges inherent to filmmaking, such as multiple characters, complex interactions, and visual cinematic effects. In order to learn to generate such content, we first create the CineVerse dataset. We use this dataset to train our proposed two-stage approach. First, we prompt a large language model (LLM) with task-specific instructions to take in a high-level scene description and generate a detailed plan for the overall setting and characters, as well as the individual shots. Then, we fine-tune a text-to-image generation model to synthesize high-quality visual keyframes. Experimental results demonstrate that CineVerse yields promising improvements in generating visually coherent and contextually rich movie scenes, paving the way for further exploration in cinematic video synthesis.

Figures (15)

• Figure 1: Cinematic scene composition. Given a simple scene description, we prompt a pre‑trained language model to generate the setting, characters with unique appearances, and detailed shot descriptions with explicit shot sizes. We then use this detailed scene plan to synthesize consistent keyframes using our fine‑tuned text‑to‑image model adapted from IC‑LoRA ic-lora specifically for our cinematic scene composition task. Compared to the baseline IC‑LoRA, our results showcase improved text‑image alignment, consistency, and continuity.
• Figure 2: Limitations of existing work. Existing multi-image text-to-image generation models struggle with complex prompts that require precise scene composition. They often fail to adhere to specified camera shots (e.g., wide, medium, close-up) and lack subject and setting consistency and continuity.
• Figure 3: Movie structure. A movie is composed of unique scenes and events that drive the storyline. Each scene consists of multiple shots establishing context, highlighting character emotions, or emphasizing key details. At the finest level, individual frames bring these shots to life. Our work aims to empower everyday users to composite cinematic scenes at the shot level.
• Figure 4: Method overview. In the stage 1, given the scene description as input, we leverage an LLM for in-context planning to produce a detailed script. This script consists of 1) Setting: A background description of the scene, 2) Characters: Individual characters with their unique appearances, and 3) Shot descriptions: The context and actions of the characters along with specified camera shots. In Stage 2, we use the generated script to synthesize multiple keyframes using text-to-image models fine-tuned on the proposed CineVerse dataset.
• Figure 5: Augumenting dataset. Storyboard20K storyboard20k scene descriptions are often ambiguous, including pronouns (e.g, him, there), making it difficult to produce keyframes with consistent scene and characters. We augment the dataset by replacing the co-reference with a specific person/object/place. We further extract detailed setting/character/shot descriptions and camera shot, forming a complete script to study cinematic scene composition.