Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

Minghan Li, Tongna Chen, Tianrui Lv, Yishuai Zhang, Suchao An, Guodong Zhou

Abstract

Existing text-to-video retrieval benchmarks are dominated by real-world footage where much of the semantics can be inferred from a single frame, leaving temporal reasoning and explicit end-state grounding under-evaluated. We introduce GenState-AI, an AI-generated benchmark centered on controlled state transitions, where each query is paired with a main video, a temporal hard negative that differs only in the decisive end-state, and a semantic hard negative with content substitution, enabling fine-grained diagnosis of temporal vs. semantic confusions beyond appearance matching. Using Wan2.2-TI2V-5B, we generate short clips whose meaning depends on precise changes in position, quantity, and object relations, providing controllable evaluation conditions for state-aware retrieval. We evaluate two representative MLLM-based baselines, and observe consistent and interpretable failure patterns: both frequently confuse the main video with the temporal hard negative and over-prefer temporally plausible but end-state-incorrect clips, indicating insufficient grounding to decisive end-state evidence, while being comparatively less sensitive to semantic substitutions. We further introduce triplet-based diagnostic analyses, including relative-order statistics and breakdowns across transition categories, to make temporal vs. semantic failure sources explicit. GenState-AI provides a focused testbed for state-aware, temporally and semantically sensitive text-to-video retrieval, and will be released on huggingface.co.

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

Abstract

Existing text-to-video retrieval benchmarks are dominated by real-world footage where much of the semantics can be inferred from a single frame, leaving temporal reasoning and explicit end-state grounding under-evaluated. We introduce GenState-AI, an AI-generated benchmark centered on controlled state transitions, where each query is paired with a main video, a temporal hard negative that differs only in the decisive end-state, and a semantic hard negative with content substitution, enabling fine-grained diagnosis of temporal vs. semantic confusions beyond appearance matching. Using Wan2.2-TI2V-5B, we generate short clips whose meaning depends on precise changes in position, quantity, and object relations, providing controllable evaluation conditions for state-aware retrieval. We evaluate two representative MLLM-based baselines, and observe consistent and interpretable failure patterns: both frequently confuse the main video with the temporal hard negative and over-prefer temporally plausible but end-state-incorrect clips, indicating insufficient grounding to decisive end-state evidence, while being comparatively less sensitive to semantic substitutions. We further introduce triplet-based diagnostic analyses, including relative-order statistics and breakdowns across transition categories, to make temporal vs. semantic failure sources explicit. GenState-AI provides a focused testbed for state-aware, temporally and semantically sensitive text-to-video retrieval, and will be released on huggingface.co.
Paper Structure (29 sections, 6 equations, 3 figures, 3 tables)

This paper contains 29 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. GenState-AI
  3. Related Work
  4. Text-to-Video Retrieval
  5. Video Retrieval Benchmarks: Real, Synthetic, and AI-Generated
  6. GenState-AI Dataset
  7. Design Principles
  8. Video Generation with Wan2.2-TI2V-5B
  9. Example Prompts with AI-Generated Videos
  10. Manual Check and Correction
  11. Evaluation Protocol
  12. Standard Retrieval Setup
  13. MLLM Baselines on GenState-AI
  14. Triplet Hard-Negative Ordering Metrics
  15. Main-vs-Temporal accuracy.
  16. ...and 14 more sections

Figures (3)

  • Figure 2: Relative-order histogram for all groups combined (195 queries total) under VCF-Lik. This provides an overall view of the model's ordering preferences across the entire dataset.
  • Figure 3: Comparison of relative-order distributions across subsets under VCF-Lik. The top panel shows absolute counts, while the bottom panel shows percentages, facilitating cross-subset analysis of ordering patterns.
  • Figure 4: Typical failure cases observed under VCF-Lik on GenState-AI. The left example shows a semantic substitution distracting the model, while the right example highlights the more common temporal confusion where a temporally plausible but end-state-incorrect video is ranked above the true match.