AI Assisted AR Assembly: Object Recognition and Computer Vision for Augmented Reality Assisted Assembly

TL;DR

The paper addresses AI-assisted AR assembly by integrating object recognition to identify parts and map them to stepwise digital guidance. It introduces an end-to-end AR workflow where a YOLOv5 model trained on synthetic data detects eight LEGO components; detections are projected into the AR space using homography for 3D bounding boxes linked to the current step. The system uses the HoloLens 2 headset and automatically advances to the next step after completion. The results demonstrate feasibility with LEGO sculptures, showing potential to eliminate manual searching and labeling. Overall, the work bridges real-time localization of physical components with contextual, stepwise AR guidance to enable AI-assisted assembly.

Abstract

We present an AI-assisted Augmented Reality assembly workflow that uses deep learning-based object recognition to identify different assembly components and display step-by-step instructions. For each assembly step, the system displays a bounding box around the corresponding components in the physical space, and where the component should be placed. By connecting assembly instructions with the real-time location of relevant components, the system eliminates the need for manual searching, sorting, or labeling of different components before each assembly. To demonstrate the feasibility of using object recognition for AR-assisted assembly, we highlight a case study involving the assembly of LEGO sculptures.

Figures (5)

• Figure 1: Object Recognition for AR Assisted Assembly
• Figure 2: System overview diagram illustrating the various software components and data -flow of the system
• Figure 3: Synthetic data generation for training object recognition algorithm
• Figure 4: Bounding Box Corresponding to Assembly Step
• Figure 5: Artifacts made using AI-Assisted AR Assembly