MotiF: Making Text Count in Image Animation with Motion Focal Loss

TL;DR

The paper tackles text-driven image-to-video (TI2V) generation, where motion following the prompt is often insufficient due to conditional image leakage. It introduces Motion Focal Loss (MotiF), which weights the diffusion training loss by a motion heatmap derived from optical flow to concentrate learning on high-motion regions, formalized as $\mathcal{L}_{\text{motif}} = \mathbb{E}_{t,\mathbf{x},\boldsymbol{\epsilon}} \| \mathbf{m}' \cdot (\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\mathbf{z}_t, \mathbf{c}, t)) \|_2^2$ with $\mathcal{L} = \mathcal{L}_{\text{diffusion}} + \lambda \mathcal{L}_{\text{motif}}$. It also introduces TI2V-Bench, a 320-image-text-pair benchmark across 22 scenarios, along with a JUICE-inspired human evaluation protocol for robust assessment of text and motion alignment. Empirical results show MotiF achieves an average 72% preference over nine open-sourced TI2V models, demonstrating improved text adherence and motion generation while remaining complementary to existing motion priors. The work provides a simple, effective, and inference-light approach to enhance TI2V performance and offers a rigorous human-centered benchmark for evaluating text-driven video animation.

Abstract

Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench and additional results are released in https://wang-sj16.github.io/motif/.

Figures (10)

• Figure 1: Motivation and results of MotiF. (a) Example video frames and the corresponding motion heatmaps calculated from optical flow. In this example, $97\%$ of the pixels are static while only $3\%$ has meaningful motion. (b) In standard TI2V training pipeline, the model may learn to over-rely on the conditional image to optimize the L2 loss. This issue has been identified in zhao2024identifying and termed as conditional image leakage. We propose MotiF to guide the model's learning to focus on regions with more motion via motion heatmap re-weighting. (c) Qualitative results comparing MotiF to the baseline on examples from our proposed TI2V-Bench evaluation set.
• Figure 2: High-level comparisons of MotiF vs. prior works. Previous TI2V generation methods mainly focused on deriving additional motion signals (motion score and/or motion mask) as inputs for the model to leverage implicitly. On the contrary, we focus on the learning objective and propose to weight the diffusion loss based on the motion intensity, that is derived from optical flow. Our method is simple, effective, and does not require additional inputs during inference. Moreover, MotiF is complementary to existing techniques.
• Figure 3: Example image-text pairs in TI2V-Bench. For each scenario (column), we first think of a scene that could be potentially animated to generate different types of motion. We include challenging scenarios when there are multiple objects (yellow/blue/red balloon) in the initial image for fine-grained control or the text prompt describes a new object (frisbee, bubbles) to enter the scene. Then we come up with different prompts and use the publicly available meta.ai tool to generate diverse sets of images. Images of low quality or those not in the appropriate initial state are removed.
• Figure 4: Human evaluation results comparing MotiF to nine open-sourced models xing2025dynamicrafterchen2023videocrafter1ma2024cinemozhang2023i2vgenchen2023seinezhao2024identifyingni2024ti2vdai2023animateanythingren2024consisti2v on TI2V-Bench. We achieved considerable improvements across the board with an average preference of $72\%$. Through examining the justification choices, we found that our model mostly excel at improving text alignment and object motion, which matches very well with our motivation.
• Figure 5: Qualitative comparison to prior works on TI2V-Bench. Sampled frames are ordered from left to right.