TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation

TL;DR

TC-Bench introduces a dedicated benchmark for temporal compositionality in video generation, addressing how attributes, relations, and backgrounds evolve over time. It provides prompts with explicit initial and final states, ground-truth videos for I2V support, and two assertion-based metrics (TCR and TC-Score) plus a frame-consistency evaluation for I2V using CLIP features. Across extensive experiments, most T2V models complete less than 20% of the described transitions, while I2V methods show higher transition completion but struggle with frame coherence, highlighting substantial gaps in temporal understanding. The work offers a foundation for rigorous temporal benchmarks and motivates future research in prompt understanding, temporal consistency, and more robust video-generation evaluation.

Abstract

Video generation has many unique challenges beyond those of image generation. The temporal dimension introduces extensive possible variations across frames, over which consistency and continuity may be violated. In this study, we move beyond evaluating simple actions and argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics. The prompts articulate the initial and final states of scenes, effectively reducing ambiguities for frame development and simplifying the assessment of transition completion. In addition, by collecting aligned real-world videos corresponding to the prompts, we expand TC-Bench's applicability from text-conditional models to image-conditional ones that can perform generative frame interpolation. We also develop new metrics to measure the completeness of component transitions in generated videos, which demonstrate significantly higher correlations with human judgments than existing metrics. Our comprehensive experimental results reveal that most video generators achieve less than 20% of the compositional changes, highlighting enormous space for future improvement. Our analysis indicates that current video generation models struggle to interpret descriptions of compositional changes and synthesize various components across different time steps.

Figures (16)

• Figure 1: Left: a common text-video pair used in video generation evaluation with no temporal compositionality. Right: a sample from our TC-Bench. Different colors of the chameleon are composed along the time axis, resulting in the vertical "edges" in the spatiotemporal image. The gap between horizontal edges shows changes in the chameleon's position and its relation with the branch.
• Figure 2: Three types of prompt-video pairs in TC-Bench. The left side shows the transition of video scene graphs. Green and blue nodes represent objects or scenes and red nodes represent attributes.
• Figure 3: Left: Assertion generation and verification covering three evaluation dimensions. Right: We investigate various methods to evaluate frame consistency for I2V models and discover that CLIP-based similarities demonstrate higher correlations with human ratings.
• Figure 4: Qualitative comparison between different models in attribute and object binding transitions.
• Figure 5: (a) Averaged CLIP cosine similarity between frame $I_k$ and the start attribute $a_1$. (b) Averaged CLIP cosine similarity between frame $I_k$ and end attribute $a_2$. (a) and (b) reflect the existence of $a_1, a_2$ as time proceeds. (c) CLIP cosine similarity between two consecutive frames.