AR as an Evaluation Playground: Bridging Metrics and Visual Perception of Computer Vision Models

TL;DR

ARCADE addresses the disconnect between numerical CV benchmarks and real-world perceptual quality by providing a capture-once-evaluate-many AR evaluation platform with pluggable model inference and interactive AR tasks. It combines reproducible data capture, real-time AR rendering, and perception-driven analysis to reveal perceptual flaws in depth and lighting models that metrics alone miss. Through a user study of 15 researchers and case studies on depth and lighting, ARCADE demonstrates improved task-level judgment, faster failure discovery, and reduced engineering effort, while also exposing latency considerations for real-time use. The work offers open-source tooling and protocols to enable principled, end-to-end evaluation of CV models in realistic AR contexts, with plans to broaden tasks and enable remote participation.

Abstract

Quantitative metrics are central to evaluating computer vision (CV) models, but they often fail to capture real-world performance due to protocol inconsistencies and ground-truth noise. While visual perception studies can complement these metrics, they often require end-to-end systems that are time-consuming to implement and setups that are difficult to reproduce. We systematically summarize key challenges in evaluating CV models and present the design of ARCADE, an evaluation platform that leverages augmented reality (AR) to enable easy, reproducible, and human-centered CV evaluation. ARCADE uses a modular architecture that provides cross-platform data collection, pluggable model inference, and interactive AR tasks, supporting both metric and visual perception evaluation. We demonstrate ARCADE through a user study with 15 participants and case studies on two representative CV tasks, depth and lighting estimation, showing that ARCADE can reveal perceptual flaws in model quality that are often missed by traditional metrics. We also evaluate ARCADE's usability and performance, showing its flexibility as a reliable real-time platform.

Figures (10)

• Figure 1: Viewing teapot placement from different angles by using ARCADE's point cloud features.
• Figure 2: (a) Outliers substantially change alignment and monotonicity. (b) A histogram of missing depth percentage in each sample from the ARKitScenes dataset.
• Figure 3: Ambient lighting sensitivity in evaluation with RealSense L515 ground truth and DepthAnythingV2 model. Opening curtains increases the error for metrics: RMSE $+11.1\%$, MAE $+14.5\%$, AbsRel $+40.3\%$.
• Figure 4: A simplified workflow of ARCADE. A scene is captured once, streamed to the configurable AR‑task engine, and rendered with two example depth models, ARKit and DepthAnythingV2 (DAv2). Researchers can visually inspect and interact with the AR tasks to iteratively design experiment protocols and perform perception evaluations.
• Figure 5: Visualization of ARCADE's features. (a) Screenshot of ARCADE's UI showing metrics alongside depth and object placement visualizations for a captured scene. (b) Illustration of the automatic virtual object re-rendering pipeline. We detect valid placement planes and place assets only on valid surfaces without re-capturing the scene on the client.