Fine-grained Controllable Video Generation via Object Appearance and Context

TL;DR

The paper tackles the challenge of fine-grained controllability in text-to-video generation by introducing FACTOR, a unified framework that injects object-level control signals (trajectory and appearance) into an existing video model via a joint encoder and adaptive cross-attention. It enables appearance control without per-subject finetuning and supports trajectory-based and appearance-based constraints through sparse inputs. Evaluations on MSR-VTT and qualitative analyses on trajectory and appearance control show improved controllability metrics (and competitive quality) over baselines, supplemented by user preferences favoring FACTOR. This approach broadens practical control over video synthesis, facilitating complex interactions and subject customization with minimal user effort.

Abstract

Text-to-video generation has shown promising results. However, by taking only natural languages as input, users often face difficulties in providing detailed information to precisely control the model's output. In this work, we propose fine-grained controllable video generation (FACTOR) to achieve detailed control. Specifically, FACTOR aims to control objects' appearances and context, including their location and category, in conjunction with the text prompt. To achieve detailed control, we propose a unified framework to jointly inject control signals into the existing text-to-video model. Our model consists of a joint encoder and adaptive cross-attention layers. By optimizing the encoder and the inserted layer, we adapt the model to generate videos that are aligned with both text prompts and fine-grained control. Compared to existing methods relying on dense control signals such as edge maps, we provide a more intuitive and user-friendly interface to allow object-level fine-grained control. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users. Extensive experiments on standard benchmark datasets and user-provided inputs validate that our model obtains a 70% improvement in controllability metrics over competitive baselines.

Figures (7)

• Figure 1: Text-to-video generation villegas2022phenaki has limited controllable ability through user-provided prompts. Even when the users augment the text prompt with additional description, the model has difficulty controlling the precise movement and appearance of the object. Text-to-Video+ControlNet zhang2023controlvideo achieves promising visual quality while requiring dense control signals extracted from a reference video. FACTOR (Ours) improves the controllability through user-friendly inputs to control the: 1) precise movement of subjects through hand-drawing trajectory and 2) visual appearance by providing reference examples.
• Figure 2: Overview. a) Joint encoder and adaptive cross-attention: a joint encoder is learned to encode the prompt and control to capture the interaction between them. The adaptive layers are inserted into the transformer blocks of the text-to-video model to take new control signals. Only the inserted layers are optimized to adapt the model to generate videos satisfying the fine-grained control. b) Condition encoding: given T time steps, the embedding of control at time $t$, $c_t$, is formed by the control for N entities, $e^n_t$, where padding tokens replace the embedding of the non-existing entity. c) Entity control: the control for entity $n$ at time $t$, $e^n_t$, is formed by the embedding of the context including the description, location, and reference appearance of the objects.
• Figure 3: Trajectory prompts. To highlight that the prompt is not enough to achieve the fine control provided by our method, we augment the text prompt to describe the trajectories specified by the bounding boxes as inputs to the text-to-video model. Phenaki villegas2022phenaki fails to generate the correct object movement, while FACTOR successfully controls the movement of generated entities with our hand-drawing trajectory input. The blue and green boxes show the location of the object in the first and last frames, respectively.
• Figure 4: Trajectory control. Given the trajectories of the two main entities in the videos as input, FACTOR brings an additional benefit to generate complex videos containing subject-object (top, middle) and subject-subject (bottom) interactions between two entities. The trajectory control inputs are omitted for simplicity.
• Figure 5: Appearance control. FACTOR generates videos with the desired object appearances. The videos contain interaction for the customized subject (left), the composition of two customized subjects (top-right), and reasonable motion of live subjects (bottom-right).