LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents

TL;DR

LangDriveCTRL tackles open-ended driving-scene editing by building an explicit, object-centric scene graph and coordinating a modular set of agents through a central Orchestrator. The system couples 3D Gaussian Splatting-based scene reconstruction with diffusion-based rendering refinements and a dedicated Trajectory/Behavior pipeline (Behavior Editing and Behavior Reviewer) to achieve high instruction alignment, structural preservation, photorealism, and realistic traffic interactions. It demonstrates near 2x improvement in alignment over prior state-of-the-art methods and provides extensive quantitative and qualitative evidence across object-level edits and multi-object behavior edits. Limitations include pedestrian editing and occasional diffusion-induced appearance changes, pointing to directions for future work and refinement.

Abstract

LangDriveCTRL is a natural-language-controllable framework for editing real-world driving videos to synthesize diverse traffic scenarios. It leverages explicit 3D scene decomposition to represent driving videos as a scene graph, containing static background and dynamic objects. To enable fine-grained editing and realism, it incorporates an agentic pipeline in which an Orchestrator transforms user instructions into execution graphs that coordinate specialized agents and tools. Specifically, an Object Grounding Agent establishes correspondence between free-form text descriptions and target object nodes in the scene graph; a Behavior Editing Agent generates multi-object trajectories from language instructions; and a Behavior Reviewer Agent iteratively reviews and refines the generated trajectories. The edited scene graph is rendered and then refined using a video diffusion tool to address artifacts introduced by object insertion and significant view changes. LangDriveCTRL supports both object node editing (removal, insertion and replacement) and multi-object behavior editing from a single natural-language instruction. Quantitatively, it achieves nearly $2\times$ higher instruction alignment than the previous SoTA, with superior structural preservation, photorealism, and traffic realism. Project page is available at: https://yunhe24.github.io/langdrivectrl/.

Figures (16)

• Figure 1: Comparison with baselines. Cosmos ali2025world achieves high visual quality but fails to align with the target behavior and modifies the background, showing poor controllability. While ChatSim wei2024editable preserves background information, it suffers from poor photorealism, inaccurate trajectory generation, and traffic violation (e.g., collision). In contrast, our method achieves photorealism, instruction alignment, structure preservation, and traffic realism simultaneously, significantly outperforming previous methods.
• Figure 2: Overall Pipeline. Given an input video and the user instruction, our pipeline first builds a scene graph, which decomposes the scene into a static background node and multiple dynamic object nodes with their trajectories. To execute the instruction, the orchestrator coordinates agents and tools from different modules to work together: the object query module localizes target object nodes in the scene graph based on text descriptions; the object node editing module performs node removal, insertion, and replacement operations; the behavior editing module generates and refines multi-object trajectories based on a feedback loop; finally, the rendering and refinement module renders the edited scene graph and refines it with a video diffusion tool. While the figure illustrates single-object editing, our pipeline is capable of multi-object editing.
• Figure 3: Qualitative comparison with baselines. The results generated by Cosmos ali2025world fail to align with the instruction and do not preserve the background well. ChatSim wei2024editable produces editing results with poor visual quality, inaccurate trajectories, and collision issues. Our method clearly outperforms them in photorealism, instruction alignment, structure preservation, and traffic realism.
• Figure 4: Qualitative editing results. We demonstrate our method's editing capabilities for diverse scenario generation. Note that for better visualization, the timestamps within each column are not strictly aligned.
• Figure 5: Ablation study for behavior feedback loop. The feedback loop effectively improves the alignment between generated trajectories and instructions while avoiding off-road behavior and collisions.