OCTO+: A Suite for Automatic Open-Vocabulary Object Placement in Mixed Reality
Aditya Sharma, Luke Yoffe, Tobias Höllerer
TL;DR
The paper tackles automatic open-vocabulary object placement in mixed reality by introducing OCTO+, a three-stage pipeline that tags possible targets, reasons over them with large-language models, and localizes natural 2D placements subsequently mapped to 3D world coordinates via raycasting. It pairs open-vocabulary vision-language components (RAM++, Grounding DINO, CLIPSeg, G-SAM) with GPT-4V/LLaVA-based reasoning, and augments realism with 3D asset generation through Shap-E for applications. The authors also present PEARL, a benchmark for Placement Evaluation of Augmented Reality Elements, and show OCTO+ achieving state-of-the-art performance across automated metrics and human evaluations, with placements deemed valid in over 70% of cases. This work offers a practical, flexible framework enabling open-vocabulary AR content insertion without fine-tuning, facilitating rapid adaptation to diverse real-world environments. It further provides a standardized evaluation protocol to drive future research in automatic AR content placement.
Abstract
One key challenge in Augmented Reality is the placement of virtual content in natural locations. Most existing automated techniques can only work with a closed-vocabulary, fixed set of objects. In this paper, we introduce and evaluate several methods for automatic object placement using recent advances in open-vocabulary vision-language models. Through a multifaceted evaluation, we identify a new state-of-the-art method, OCTO+. We also introduce a benchmark for automatically evaluating the placement of virtual objects in augmented reality, alleviating the need for costly user studies. Through this, in addition to human evaluations, we find that OCTO+ places objects in a valid region over 70% of the time, outperforming other methods on a range of metrics.