Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation

Abstract

A major challenge in autonomous driving is the "long tail" of safety-critical edge cases, which often emerge from unusual combinations of common traffic elements. Synthesizing these scenarios is crucial, yet current controllable generative models provide incomplete or entangled guidance, preventing the independent manipulation of scene structure, object identity, and ego actions. We introduce CompoSIA, a compositional driving video simulator that disentangles these traffic factors, enabling fine-grained control over diverse adversarial driving scenarios. To support controllable identity replacement of scene elements, we propose a noise-level identity injection, allowing pose-agnostic identity generation across diverse element poses, all from a single reference image. Furthermore, a hierarchical dual-branch action control mechanism is introduced to improve action controllability. Such disentangled control enables adversarial scenario synthesis-systematically combining safe elements into dangerous configurations that entangled generators cannot produce. Extensive comparisons demonstrate superior controllable generation quality over state-of-the-art baselines, with a 17% improvement in FVD for identity editing and reductions of 30% and 47% in rotation and translation errors for action control. Furthermore, downstream stress-testing reveals substantial planner failures: across editing modalities, the average collision rate of 3s increases by 173%.

Figures (11)

• Figure 1: CompoSIA is a powerful simulator for synthesizing rare driving scenes. While existing driving world models focus on generating, we enables explicit control over structure, identity, and ego action.
• Figure 2: Overview of the CompoSIA framework. During training (top), structure, identity, and ego-action signals are explicitly decomposed and injected into a Flow Matching–based DiT backbone in a disentangled yet compositional manner. During sampling (bottom), different combinations of conditions enable controllable driving video generation, supporting multiple editing applications.
• Figure 3: Design of hierarchical dual-branch action Conditioning. Local residual modulation accelerates early convergence, while global PRoPE embedding improves overall accuracy.
• Figure 4: Identity editing for the first frame. We preserve the background to anchor scene identity while editing the foreground. If the reference area is smaller than the original area, the intermediate region is treated as an inpainting area during denoising.
• Figure 5: Comparison of generation quality.CompoSIA renders vehicles and pedestrians with more structurally coherent geometry, while maintaining stronger cross-view and temporal consistency.