MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent

TL;DR

MotionAgent addresses the challenge of fine-grained, text-driven control for image-to-video generation. It introduces a motion field agent that converts textual motion descriptions into explicit object trajectories and camera extrinsics, and uses an analytical optical flow composition to fuse these into unified flow maps, which are then used by a tuned optical flow adapter on a base diffusion model. A rethinking loop provides feedback to refine trajectories and camera motion, improving alignment with prompts and reducing artifacts. Evaluations on VBench and a purpose-built motion-alignment benchmark show state-of-the-art performance in motion control and competitive video quality, with robust results across different LLM backbones and prompt complexities, enabling precise, text-driven video synthesis without large motion-annotated datasets.

Abstract

We propose MotionAgent, enabling fine-grained motion control for text-guided image-to-video generation. The key technique is the motion field agent that converts motion information in text prompts into explicit motion fields, providing flexible and precise motion guidance. Specifically, the agent extracts the object movement and camera motion described in the text and converts them into object trajectories and camera extrinsics, respectively. An analytical optical flow composition module integrates these motion representations in 3D space and projects them into a unified optical flow. An optical flow adapter takes the flow to control the base image-to-video diffusion model for generating fine-grained controlled videos. The significant improvement in the Video-Text Camera Motion metrics on VBench indicates that our method achieves precise control over camera motion. We construct a subset of VBench to evaluate the alignment of motion information in the text and the generated video, outperforming other advanced models on motion generation accuracy.

Figures (14)

• Figure 1: Different frameworks of I2V generation models. (a) Controllable I2V generation via text encoder. (b) Controllable I2V generation via special control module. (c) Our method, controllable I2V generation via motion field agent.
• Figure 2: Pipeline of Motion Field Agent. Step 1: the agent first parses the input text, dividing the motion information into two parts that respectively describe object movement and camera motion. Step 2: the agent draws the object trajectories according to the text of object movement. Step 3: the agent directly generates the camera extrinsics based on the text of camera motion. Step 4 (optional): the agent rethinks and corrects the former actions according to the generated video.
• Figure 3: Controllable I2V Generation Model. (a) The analytical optical flow composition module calculates unified optical flow maps based on the object trajectories and camera extrinsics. (b) The unified flow maps are fed into a fine-tuned optical flow adapter as the control condition. Then, we generate precisely controlled video results based on a base I2V diffusion model.
• Figure 4: Comparison results of controllable I2V generation on our benchmark. The motion described in the text is in bold.
• Figure 5: Fine-grained controllable video results generated by our method. Lines 1-2 are multiple object movements control; Line 3 is camera motion control.