Semantic-Aware Label Placement for Augmented Reality in Street View

TL;DR

The paper tackles label placement in AR street-view scenarios, aiming to keep labels readable while avoiding occlusion of important real-world content. It introduces a Guidance Map that fuses saliency, semantic segmentation, and a learned task-prior, learned from a manually labeled dataset. An optimization framework combines label and leader-line energies to compute label layouts, with weights learned from data, and a greedy placement strategy is used for efficiency. Experiments on Cityscapes-derived data demonstrate improvements over baselines in alignment with human preferences and reduced occlusion, highlighting the method's potential for practical AR navigation and broader AR applications.

Abstract

In an augmented reality (AR) application, placing labels in a manner that is clear and readable without occluding the critical information from the real-world can be a challenging problem. This paper introduces a label placement technique for AR used in street view scenarios. We propose a semantic-aware task-specific label placement method by identifying potentially important image regions through a novel feature map, which we refer to as guidance map. Given an input image, its saliency information, semantic information and the task-specific importance prior are integrated into the guidance map for our labeling task. To learn the task prior, we created a label placement dataset with the users' labeling preferences, as well as use it for evaluation. Our solution encodes the constraints for placing labels in an optimization problem to obtain the final label layout, and the labels will be placed in appropriate positions to reduce the chances of overlaying important real-world objects in street view AR scenarios. The experimental validation shows clearly the benefits of our method over previous solutions in the AR street view navigation and similar applications.

Figures (8)

• Figure 1: Flowchart of the proposed algorithm
• Figure 2: Schematic diagram for AR navigation display in AR-HUD
• Figure 3: Illustration for Manual Label Placement dataset collecting. Participants view the image with 8 labels, and use a mouse to drag the labels onto the appropriate place, just like the right image.
• Figure 4: Visual comparison of selected saliency detection.
• Figure 5: An incorrectly calculated example (left) and the implemented method of calculation (right).