CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluation

Abstract

Generating safety-critical driving scenarios requires understanding why dangerous interactions arise, rather than merely forcing collisions. However, existing methods rely on heuristic adversarial agent selection and unstructured perturbations, lacking explicit modeling of interaction dependencies and thus exhibiting a realism--adversarial trade-off. We present CounterScene, a framework that endows closed-loop generative BEV world models with structured counterfactual reasoning for safety-critical scenario generation. Given a safe scene, CounterScene asks: what if the causally critical agent had behaved differently? To answer this, we introduce causal adversarial agent identification to identify the critical agent and classify conflict types, and develop a conflict-aware interactive world model in which a causal interaction graph is used to explicitly model dynamic inter-agent dependencies. Building on this structure, stage-adaptive counterfactual guidance performs minimal interventions on the identified agent, removing its spatial and temporal safety margins while allowing risk to emerge through natural interaction propagation. Extensive experiments on nuScenes demonstrate that CounterScene achieves the strongest adversarial effectiveness while maintaining superior trajectory realism across all horizons, improving long-horizon collision rate from 12.3% to 22.7% over the strongest baseline with better realism (ADE 1.88 vs.2.09). Notably, this advantage further widens over longer rollouts, and CounterScene generalizes zero-shot to nuPlan with state-of-the-art realism.

Figures (7)

• Figure 1: Counterfactual causal reasoning for safety-critical generative BEV world model. Given an observed traffic scene where the critical agent waits and the ego vehicle passes safely, we ask the counterfactual question: What if the critical agent did not wait? By intervening on the agent trajectory, CounterScene constructs a counterfactual world that induces safety-critical interactions while preserving realistic traffic dynamics. This is achieved through causal adversarial agent identification, causal interaction graph modeling, and counterfactual guidance in diffusion.
• Figure 2: Overview of CounterScene. The framework consists of four modules. (1) Causal adversarial agent selection identifies the most critical interacting agent using geometric conflict analysis and danger scoring. (2) Causal Interaction Graph (CIG) encodes conflict-aware interaction relationships among agents. (3) A diffusion-based interactive BEV world model with a SceneTransformer denoiser generates multi-agent trajectories. (4) Counterfactual guidance perturbs the adversarial agent trajectory during denoising to construct safety-critical interactions while maintaining realistic traffic dynamics.
• Figure 3: Qualitative comparison in a merging scenario (Scene 103, 10-second closed-loop rollout at 10 Hz).Factual (Original): In the recorded safe scene, the merging vehicle from the side road maintains a temporal safety margin by yielding, allowing the ego vehicle to pass the intersection safely. CounterScene (Ours): By causally identifying the yielding vehicle and progressively compressing its spatiotemporal margin, CounterScene generates a highly realistic safety-critical counterfactual: the adversarial vehicle aggressively cuts into the main road, forcing the ego vehicle into an emergency hard brake (matching our high HBR metric) to avoid a severe collision. Baselines: In contrast, baseline methods fail to capture the causal interaction structure. They either preserve the safe yielding behavior (failing to induce adversarial risk) or exhibit varying degrees of unstructured trajectory distortion, highlighting their inability to generate structurally valid and physically plausible critical interactions.
• Figure 4: Qualitative comparison of a merging collision in Scene 110. Factual (Original): In the recorded scene, the merging vehicle from the side road actively maintains safety by yielding, providing a sufficient spatiotemporal margin for the ego vehicle to pass. CounterScene (Ours): By accurately identifying the yielding vehicle as the causal variable and surgically stripping its safety margin, CounterScene generates a highly realistic collision. The adversarial agent aggressively merges into the main road without yielding, resulting in a physically plausible crash while the rest of the scene evolves naturally. Baselines: Lacking structural causal reasoning, baseline methods completely fail to generate meaningful adversarial pressure. They either passively preserve the original yielding behavior (resulting in zero risk) or introduce unstructured trajectory noise that fails to culminate in a realistic collision.
• Figure 5: Qualitative comparison of a rear-end collision in Scene 905. Factual (Original): In the recorded safe scene, the trailing vehicle actively maintains a safe longitudinal distance, preserving a temporal margin that prevents a collision as the ego vehicle navigates the lane. CounterScene (Ours): By causally identifying the trailing vehicle and surgically compressing its longitudinal safety margin, our framework generates a highly realistic rear-end crash. The adversarial agent aggressively accelerates into the ego vehicle, demonstrating how targeted temporal compression naturally evolves into a high-risk scenario without distorting the surrounding traffic dynamics. Baselines: Lacking structured causal reasoning, baseline methods fail to capture the nature of this longitudinal interaction. They either maintain the original safe following distance (generating zero adversarial pressure) or apply unstructured perturbations that cause the vehicle to veer off-course, ultimately failing to induce a physically plausible rear-end collision.