Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble

TL;DR

This work addresses automated evaluation of AR-generated scenes by leveraging Vision-Language Models (VLMs) and introduces the DiverseAR dataset to benchmark perception and description capabilities. It evaluates three state-of-the-art VLMs (GPT-4o, Gemini, Claude) using two prompt regimes across three AR scene complexity levels, defining perception and description metrics $TPR_P$, $TPR_D$, and $TNR_P$. Key findings show VLMs can reliably identify obvious AR content ($TPR_P$ up to 0.93) and describe it ($TPR_D$ up to 0.71) but falter on seamlessly integrated scenes, with performance degrading as complexity rises; GPT and Gemini exhibit complementary strengths depending on the task and prompt, while Claude lags behind. The study also reports a human user study that mostly aligns with VLM trends but reveals higher accuracy for humans on harder scenes, motivating AR-specific fine-tuning and temporal AR-content evaluation to better align automated assessments with human perception.

Abstract

Augmented Reality (AR) enhances the real world by integrating virtual content, yet ensuring the quality, usability, and safety of AR experiences presents significant challenges. Could Vision-Language Models (VLMs) offer a solution for the automated evaluation of AR-generated scenes? Could Vision-Language Models (VLMs) offer a solution for the automated evaluation of AR-generated scenes? In this study, we evaluate the capabilities of three state-of-the-art commercial VLMs -- GPT, Gemini, and Claude -- in identifying and describing AR scenes. For this purpose, we use DiverseAR, the first AR dataset specifically designed to assess VLMs' ability to analyze virtual content across a wide range of AR scene complexities. Our findings demonstrate that VLMs are generally capable of perceiving and describing AR scenes, achieving a True Positive Rate (TPR) of up to 93% for perception and 71% for description. While they excel at identifying obvious virtual objects, such as a glowing apple, they struggle when faced with seamlessly integrated content, such as a virtual pot with realistic shadows. Our results highlight both the strengths and the limitations of VLMs in understanding AR scenarios. We identify key factors affecting VLM performance, including virtual content placement, rendering quality, and physical plausibility. This study underscores the potential of VLMs as tools for evaluating the quality of AR experiences.

Figures (9)

• Figure 1: AR content is integrated into real-world contexts: a virtual pot placed on a cat (top) and a virtual apple placed on a table (bottom). GPT can accurately recognize and describe the AR pot on the cat but mistakenly identifies the real toys as AR elements.
• Figure 2: Examples of key characteristics of the virtual and real objects in the DiverseAR dataset. a) Virtual pot positioned in an abnormal place (on a cat); b) Real and virtual apples positioned in a typical place (table); c) Virtual toy intersecting a real palm; d) Floating virtual cup; e) Virtual toy with unusual size; f) Virtual sign in an abnormal pose; g) Large virtual toy on the table with its shadow cast in the wrong direction; h) Virtual laptop with no shadow; i) Virtual apple alongside real objects under bright lighting; j) Three virtual cans alongside real objects under dim lighting; k) Transparent, glowing virtual highlights; l) Low-render-quality virtual basket.
• Figure 3: The True Positive Rate for Perception ($TPR_P$) of the three VLMs across varying complexity levels using G (general) and T (task-aware) image captioning prompts. VLMs show a consistent decline in performance as AR scene complexity increases.
• Figure 4: The True Positive Rate for Description ($TPR_D$) of the three VLMs across varying AR scene complexity levels using G (general) and T (task-aware) image captioning prompts. Notably, GPT outperforms both Gemini and Claude on medium and hard AR scenes using prompt T.
• Figure 5: These images feature an AR-generated "Exit" sign integrated into real-world contexts. With the prompt T, GPT accurately identifies the sign with a QR code (bottom) but misclassifies it as a real object without the code (top).