MIRAGE: Enabling Real-Time Automotive Mediated Reality

TL;DR

MIRAGE addresses the sim-to-real gap in Automotive Mediated Reality by delivering a holistic, open-source pipeline that runs real-time AR, DR, and ModR effects in a vehicle using state-of-the-art computer vision models (e.g., YOLO for segmentation, DepthAnything for depth, MI-GAN for inpainting) within Unity via the Unity Inference Engine. The authors implement 15 AMR effects, demonstrate a windshield-display style WSD via head-mounted passthrough, and conduct an expert user study (N=9) to assess usability, workload, and perceived usefulness, revealing both promise and practical challenges such as latency, UI usability, and artifacts. Key contributions include a taxonomy of AMR effects, a modular, real-time in-vehicle pipeline, an open-source repository, and empirical expert feedback informing ethics, interaction design, and future research. The work establishes MIRAGE as a practical platform for prototyping and evaluating AMR concepts in real traffic, enabling earlier identification of perceptual, technical, and ethical issues beyond simulator environments, and guiding subsequent human-centered studies and safety considerations.

Abstract

Traffic is inherently dangerous, with around 1.19 million fatalities annually. Automotive Mediated Reality (AMR) can enhance driving safety by overlaying critical information (e.g., outlines, icons, text) on key objects to improve awareness, altering objects' appearance to simplify traffic situations, and diminishing their appearance to minimize distractions. However, real-world AMR evaluation remains limited due to technical challenges. To fill this sim-to-real gap, we present MIRAGE, an open-source tool that enables real-time AMR in real vehicles. MIRAGE implements 15 effects across the AMR spectrum of augmented, diminished, and modified reality using state-of-the-art computational models for object detection and segmentation, depth estimation, and inpainting. In an on-road expert user study (N=9) of MIRAGE, participants enjoyed the AMR experience while pointing out technical limitations and identifying use cases for AMR. We discuss these results in relation to prior work and outline implications for AMR ethics and interaction design.

Figures (9)

• Figure 1: Automotive Situation Awareness–Appraisal Mediation Cycle. AMR effects (Modify, Augment, Diminish) alter the visual scene before perception, shaping SA across Levels 1–3 endsley2017toward. SA influences behavior and attention in a fast, moment-to-moment manner endsley2017toward, whereas downstream appraisal (e.g., trust lee2004trust, perceived safety carsten2019can, enjoyment in non-driving tasks hock_carvr_2017schramm2025augmented) shapes behavior and attention more slowly and evaluatively scherer2001appraisal. Appraisal is further informed by expectations, prior experience, goals, and affective states scherer2001appraisal, although these influences are not explicitly visualized in the cycle. Behavior and attention affect subsequent SA endsley2017toward. The mediation cycle applies across automation levels (drivers in SAE 0–2 manual driving; passengers in SAE 3–5 automated driving), where AMR may support different driving or non-driving tasks.
• Figure 2: AMR includes AR, DR, and ModR approaches. AR adds information to the driving environment and expands information on specific objects. DR reduces the visual information of objects, making them less detailed. ModR modifies objects by translating, rotating, scaling, changing their art style, and replacing them entirely.
• Figure 3: Overview of MIRAGE. (a) A virtual WSD displays AMR effects. In this example, a blue outline is added to cars, pedestrians are colored red with icons above their heads, and traffic lights have yellow bounding boxes. The input image for the pipeline is recorded by a forward-facing camera (b). The user sees the WSD through an HMD (c). AMR effects can be controlled via UI (d) using a gamepad (e). The driver and experimenter were present during the expert user study (see Section \ref{['ch:study']}).
• Figure 4: MIRAGE processing pipeline. The input image is first pre-processed for semantic segmentation using YOLO11 and depth estimation using DepthAnythingV2. Next, an object removal stage computes a selective inpainting mask, and the MI-GAN model generates replacement pixels for targeted objects. Simultaneously, post-processing algorithms create the selected visualizations from the input and pre-processed data. Finally, all outputs are combined with the original image.
• Figure 5: Overview of the 15 AMR effects we implemented for MIRAGE applied to a car and traffic light in real-time. The effects Augment, Diminish, and Modify the visual appearance of these objects.