Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Reason-then-Describe Instruction Interpreter for Controllable Video Generation

Shengqiong Wu, Weicai Ye, Yuanxing Zhang, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Kun Gai, Hao Fei, Tat-Seng Chua

TL;DR

ReaDe addresses the gap between concise user prompts and the detailed prompts required by modern video generators by introducing a universal, model-agnostic instruction interpreter that follows a reason-then-describe paradigm. Built on a multimodal LLM with a camera encoder, ReaDe undergoes a two-stage training regimen: (i) CoT-based reasoning initialization with supervised fine-tuning to produce structured prompts, and (ii) reinforcement learning using a multi-dimensional reward and GRPO to refine reasoning and caption quality. The approach yields consistent improvements in instruction fidelity, caption accuracy, and downstream video quality across single- and multi-condition scenarios, with strong generalization to unseen, reasoning-intensive inputs. These results demonstrate ReaDe’s potential to align controllable video generation more closely with user intent, enabling more faithful, coherent, and flexible video synthesis in practice.

Abstract

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.

A Reason-then-Describe Instruction Interpreter for Controllable Video Generation

TL;DR

ReaDe addresses the gap between concise user prompts and the detailed prompts required by modern video generators by introducing a universal, model-agnostic instruction interpreter that follows a reason-then-describe paradigm. Built on a multimodal LLM with a camera encoder, ReaDe undergoes a two-stage training regimen: (i) CoT-based reasoning initialization with supervised fine-tuning to produce structured prompts, and (ii) reinforcement learning using a multi-dimensional reward and GRPO to refine reasoning and caption quality. The approach yields consistent improvements in instruction fidelity, caption accuracy, and downstream video quality across single- and multi-condition scenarios, with strong generalization to unseen, reasoning-intensive inputs. These results demonstrate ReaDe’s potential to align controllable video generation more closely with user intent, enabling more faithful, coherent, and flexible video synthesis in practice.

Abstract

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.

Paper Structure

This paper contains 43 sections, 6 equations, 13 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Stage-1: CoT-guided Reasoning Initialization
  5. CoT Data Construction.
  6. SFT-based Optimization.
  7. Stage-2: Incentive Reinforcement Learning
  8. Reward Assignment.
  9. Answer Structure Reward.
  10. Multi-dimensional Content Reward.
  11. Data Construction for RL Learning
  12. Optimization via GRPO
  13. Experiments
  14. Settings
  15. Main Results
  16. ...and 28 more sections

Figures (13)

  • Figure 1: Text-only prompt optimizers and data-hungry multimodal methods (e.g., Any2Caption) remain brittle, performing poorly on reasoning-intensive and unseen instructions.
  • Figure 2:
  • Figure 3: Generalization capability of ReaDe. The heatmap shows the dense caption intention accuracy under different training–evaluation condition pairs. The y-axis corresponds to training conditions, while the x-axis denotes evaluation conditions.
  • Figure 4: Illustration of prompt optimization for raw prompts. The left panel shows text-to-image generation results produced by CogVideoX-2B, while the right panel presents image-to-text generation results obtained with CogVideoX-5B-I2V. Some prompts are omitted due to space constraints.
  • Figure 5: Qualitative comparison of the generation quality across original prompts, interpreted prompts by Any2Caption, and ReaDe. The first two rows are generated via Kling1.6, the third is generated via FullDiT, and the last one is generated with SketchVideo.
  • ...and 8 more figures