EVER: Edge-Assisted Auto-Verification for Mobile MR-Aided Operation

TL;DR

The paper tackles the challenge of real-time, reliable auto-verification in mobile MR-aided operations, where traditional frame similarity fails due to discrepancies between virtual guidance and physical objects. It introduces EVER, an edge-assisted framework that combines motion-triggered frame capture, rendering-informed segmentation, and an IoU-based threshold to verify user actions, with compute offloaded to an edge server for latency and energy efficiency. Extensive evaluations on synthetic PCB/breadboard datasets and a custom LEGO dataset show EVER achieving over 90% verification accuracy with end-to-end latency under 100 ms, far below typical human reaction times. Key contributions include a unified end-to-end design, a segmentation-based verification pipeline, tag-based localization, frame alignment optimizations, and comprehensive benchmarking across codecs, resolutions, and similarity metrics, demonstrating practical deployment potential in MR-aided operations.

Abstract

Mixed Reality (MR)-aided operation overlays digital objects on the physical world to provide a more immersive and intuitive operation process. A primary challenge is the precise and fast auto-verification of whether the user follows MR guidance by comparing frames before and after each operation. The pre-operation frame includes virtual guiding objects, while the post-operation frame contains physical counterparts. Existing approaches fall short of accounting for the discrepancies between physical and virtual objects due to imperfect 3D modeling or lighting estimation. In this paper, we propose EVER: an edge-assisted auto-verification system for mobile MR-aided operations. Unlike traditional frame-based similarity comparisons, EVER leverages the segmentation model and rendering pipeline adapted to the unique attributes of frames with physical pieces and those with their virtual counterparts; it adopts a threshold-based strategy using Intersection over Union (IoU) metrics for accurate auto-verification. To ensure fast auto-verification and low energy consumption, EVER offloads compute-intensive tasks to an edge server. Through comprehensive evaluations of public datasets and custom datasets with practical implementation, EVER achieves over 90% verification accuracy within 100 milliseconds (significantly faster than average human reaction time of approximately 273 milliseconds), while consuming only minimal additional computational resources and energy compared to a system without auto-verification.

Figures (14)

• Figure 1: System architecture.
• Figure 2: Example frames correctly following MR instructions.
• Figure 3: Example of frame alignment.
• Figure 4: LEGO models used in experiments.
• Figure 5: Prototype of our system.