LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment

TL;DR

LiFT addresses the challenge of aligning text-to-video generation with subjective human preferences by introducing a three-stage pipeline: LiFT-HRA for richly annotated human feedback, LiFT-Critic to learn a reward function that captures reasoning behind judgments, and reward-weighted fine-tuning to align T2V outputs with human expectations. The approach emphasizes video-specific temporal criteria (semantic consistency, motion smoothness, video fidelity) and demonstrates that fine-tuning CogVideoX-2B with LiFT yields superior performance across 16 metrics compared to a larger baseline. Ablation studies show the value of reason-based annotations, larger reward models, and incorporating real video data for temporal fidelity. Overall, LiFT highlights the practical potential of human-feedback-driven alignment for high-quality, human-aligned video synthesis and sets the stage for broader, ethically mindful deployment of T2V systems.

Abstract

Recent advances in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are subjective and challenging to formalize as objective functions. Existing studies train video quality assessment models that rely on human-annotated ratings for video evaluation but overlook the reasoning behind evaluations, limiting their ability to capture nuanced human criteria. Moreover, aligning T2V model using video-based human feedback remains unexplored. Therefore, this paper proposes LiFT, the first method designed to leverage human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.

Figures (16)

• Figure 1: An illustration of the proposed LiFT. First, we construct a comprehensive human feedback dataset. Then, a reward model is trained to learn the reward function. Finally, the T2V model is fine-tuned by the reward model to align its output with human expectations.
• Figure 2: The overview of our proposed pipeline. This illustration depicts three key steps of our fine-tuning pipeline: (1) Human Feedback Collection: we generate video-text pairs using prompts expanded from random category words with an LLM, then annotate them to create LiFT-HRA. (2) Reward Function Learning: a visual-language model LiFT-Critic, is trained to predict human preference scores across three dimensions, learning the reward function from the dataset. (3) T2V Model Alignment: LiFT-Critic evaluates the T2V-generated videos, assigns scores, and maps them into a reward weight to fine-tune the T2V model, aligning it with human preferences.
• Figure 3: An illustration of our annotation UI. Annotators evaluate each video by assigning scores to each dimension and providing the reason behind their assessments.
• Figure 4: The visualized statistic results of our proposed LiFT-HRA. It illustrates the distribution of category types, the video count across these categories, and the corresponding human feedback distribution for each category.
• Figure 5: Qualitative Comparison. We compare the performance of our CogVideoX-2B-LiFT (fine-tuned using reward-weighted learning) against CogVideoX-2B and CogVideoX-5B.