Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments

Junfeng Lin, Yanming Xiu, Maria Gorlatova

TL;DR

This work investigates how user prompts influence open-set object detection (OSOD) in XR settings, addressing robustness gaps when prompts are ambiguous or overly detailed. It evaluates GroundingDINO and YOLO-E with vision-language model–generated prompts of varying specificity and tests two prompt-enhancement strategies (Key Object Extraction and Semantic Category Grounding) to refine inputs before detection. Findings show that both models are stable under underdetailed and standard prompts but struggle with pragmatic ambiguity, while overdetailed prompts mainly hinder GroundingDINO; enhancement strategies substantially improve robustness under ambiguity, with notable gains in localization and confidence. The study offers practical prompt design guidelines and enhancement methods to improve OSOD reliability in XR, enabling more dependable perception in AR/VR applications.

Abstract

Open-set object detection (OSOD) localizes objects while identifying and rejecting unknown classes at inference. While recent OSOD models perform well on benchmarks, their behavior under realistic user prompting remains underexplored. In interactive XR settings, user-generated prompts are often ambiguous, underspecified, or overly detailed. To study prompt-conditioned robustness, we evaluate two OSOD models, GroundingDINO and YOLO-E, on real-world XR images and simulate diverse user prompting behaviors using vision-language models. We consider four prompt types: standard, underdetailed, overdetailed, and pragmatically ambiguous, and examine the impact of two enhancement strategies on these prompts. Results show that both models exhibit stable performance under underdetailed and standard prompts, while they suffer degradation under ambiguous prompts. Overdetailed prompts primarily affect GroundingDINO. Prompt enhancement substantially improves robustness under ambiguity, yielding gains exceeding 55% mIoU and 41% average confidence. Based on the findings, we propose several prompting strategies and prompt enhancement methods for OSOD models in XR environments.

User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments

TL;DR

This work investigates how user prompts influence open-set object detection (OSOD) in XR settings, addressing robustness gaps when prompts are ambiguous or overly detailed. It evaluates GroundingDINO and YOLO-E with vision-language model–generated prompts of varying specificity and tests two prompt-enhancement strategies (Key Object Extraction and Semantic Category Grounding) to refine inputs before detection. Findings show that both models are stable under underdetailed and standard prompts but struggle with pragmatic ambiguity, while overdetailed prompts mainly hinder GroundingDINO; enhancement strategies substantially improve robustness under ambiguity, with notable gains in localization and confidence. The study offers practical prompt design guidelines and enhancement methods to improve OSOD reliability in XR, enabling more dependable perception in AR/VR applications.

Abstract

Open-set object detection (OSOD) localizes objects while identifying and rejecting unknown classes at inference. While recent OSOD models perform well on benchmarks, their behavior under realistic user prompting remains underexplored. In interactive XR settings, user-generated prompts are often ambiguous, underspecified, or overly detailed. To study prompt-conditioned robustness, we evaluate two OSOD models, GroundingDINO and YOLO-E, on real-world XR images and simulate diverse user prompting behaviors using vision-language models. We consider four prompt types: standard, underdetailed, overdetailed, and pragmatically ambiguous, and examine the impact of two enhancement strategies on these prompts. Results show that both models exhibit stable performance under underdetailed and standard prompts, while they suffer degradation under ambiguous prompts. Overdetailed prompts primarily affect GroundingDINO. Prompt enhancement substantially improves robustness under ambiguity, yielding gains exceeding 55% mIoU and 41% average confidence. Based on the findings, we propose several prompting strategies and prompt enhancement methods for OSOD models in XR environments.
Paper Structure (17 sections, 3 equations, 5 figures, 1 table)

This paper contains 17 sections, 3 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open-set Object Detection in XR Environment
  4. Vision Language Models
  5. Problem Formulation: Prompt-Conditioned Robustness in XR
  6. Open-Set Detection with Language Conditioning
  7. Failure Patterns
  8. Prompt-Conditioned Robustness in XR
  9. Experiment Setup
  10. Dataset
  11. Evaluation Pipeline
  12. Evaluation Metric
  13. Results
  14. Results on Datasets with Simulated Prompts
  15. Failure Case Analysis
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Example of open-set object detection. Given the text prompt “dinosaur,” the model detects a plush dinosaur: an object category uncommon in standard object detection benchmarks.
  • Figure 2: Example images from the DiverseAR and DiverseAR+ datasets, with the target objects indicated by bounding boxes: (a) a bottle on the bedroom floor; (b) a pair of scissors on a kitchen table; (c) two bottles of water on a table in an office environment.
  • Figure 3: Architecture of the evaluation pipeline.
  • Figure 4: Examples of the initial prompts for OSOD models, generated by VLM: underdetailed, standard, overdetailed, ambiguous.
  • Figure 5: Representative cases of OSOD under different prompt formulations in AR scenes. (a) Pragmatic ambiguity prompt + no enhancement + GD; (b) Pragmatic ambiguity prompt + semantic category grounding + GD; (c) Overdetailed prompt + no enhancement + YOLO-E; (d) Overdetailed prompt + no enhancement + GD; (e) Overdetailed prompt + semantic category grounding + YOLO-E; (f) Overdetailed prompt + semantic category grounding + GD.