TGT: Text-Grounded Trajectories for Locally Controlled Video Generation

TL;DR

Text-Grounded Trajectories (TGT) addresses the lack of fine-grained control in text-to-video generation by grounding point trajectories to localized text descriptions. It introduces Location-Aware Cross-Attention (LACA), a lightweight plug-in that aligns local trajectory-associated text with nearby visual tokens while preserving global prompt adherence via a dual-CFG scheme. A scalable data pipeline creates trajectory–text supervision by grounding entities in frames and propagating points into full trajectories, resulting in a large two-million-clip corpus. Empirical results show improved trajectory accuracy and local text alignment without sacrificing visual quality, enabling practical applications such as video-to-video mirroring and targeted local editing. Overall, TGT demonstrates that explicit per-trajectory grounding can substantially enhance controllability in multi-object video generation with realistic temporal coherence.

Abstract

Text-to-video generation has advanced rapidly in visual fidelity, whereas standard methods still have limited ability to control the subject composition of generated scenes. Prior work shows that adding localized text control signals, such as bounding boxes or segmentation masks, can help. However, these methods struggle in complex scenarios and degrade in multi-object settings, offering limited precision and lacking a clear correspondence between individual trajectories and visual entities as the number of controllable objects increases. We introduce Text-Grounded Trajectories (TGT), a framework that conditions video generation on trajectories paired with localized text descriptions. We propose Location-Aware Cross-Attention (LACA) to integrate these signals and adopt a dual-CFG scheme to separately modulate local and global text guidance. In addition, we develop a data processing pipeline that produces trajectories with localized descriptions of tracked entities, and we annotate two million high quality video clips to train TGT. Together, these components enable TGT to use point trajectories as intuitive motion handles, pairing each trajectory with text to control both appearance and motion. Extensive experiments show that TGT achieves higher visual quality, more accurate text alignment, and improved motion controllability compared with prior approaches. Website: https://textgroundedtraj.github.io.

Figures (11)

• Figure 1: TGT generates videos following input trajectories with each trajectory associated with user specified local text prompt. Left: the user input trajectories or static points. Right: generated videos with trajectory visualizations, large dot indicate point location on the cut frame.
• Figure 2: Pipeline of TGT. We propose Location-Aware Cross-Attention (LACA), a cross-attention that injects entity and location information into every DiT block’s forward pass. LACA performs masked attention, ensuring each visual token attends only to its corresponding local text token.
• Figure 3: TGT data collection pipeline. We first apply Grounded SAM to segment entities and select representative points on every entity. These points are then passed to a point tracker and a distilled VLM to extract trajectories and localized captions, yielding paired trajectory–text supervision.
• Figure 4: Qualitative comparison between TGT and baseline methods. Input point locations are labeled in color on each video frame. Local text prompts are shown on the left. Our method generates more aligned and realistic entity that follows the local text and achieves greater precision in motion.
• Figure 5: Practical applications of TGT. We extract dense motion trajectories and local captions from a reference video and use them as conditions to generate a new video. The left example shows video-to-video mirroring, while the right demonstrates text-driven local editing.