Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality

Alexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, Jenny Sabin

TL;DR

The paper tackles adaptive, perception-driven AR guidance for physical assembly by combining real-time object recognition with hand tracking to create a digital twin and overlay actionable, step-specific instructions. It introduces a synthetic-data training pipeline for a YOLOv5 detector to recognize custom components, and integrates a parametric assembly model with constraint-based optimization and structural analysis to dynamically update instructions in response to user actions. Demonstrations with LEGO and 3D-printed parts show substantial reductions in assembly time and high interaction accuracy, illustrating how deviations can become opportunities for exploration rather than errors. Overall, the work advances embodied making by tightly coupling perception, feedback, and adaptive guidance in AR to support creative, iterative assembly processes.

Abstract

Recent advances in augmented reality (AR) have enabled interactive systems that assist users in physical assembly tasks. In this paper, we present an AR-assisted assembly workflow that leverages object recognition and hand tracking to (1) identify custom components, (2) display step-by-step instructions, (3) detect assembly deviations, and (4) dynamically update the instructions based on users' hands-on interactions with physical parts. Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace. For each assembly step, it overlays bounding boxes in AR to indicate both the current position and the target placement of relevant components, while hand-tracking data verifies whether the user interacts with the correct part. Rather than enforcing a fixed sequence, the system highlights potential assembly errors and interprets user deviations as opportunities for iteration and creative exploration. A case study with LEGO blocks and custom 3D-printed components demonstrates how the system links digital instructions to physical assembly, eliminating the need for manual searching, sorting, or labeling of parts.

Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality

TL;DR

The paper tackles adaptive, perception-driven AR guidance for physical assembly by combining real-time object recognition with hand tracking to create a digital twin and overlay actionable, step-specific instructions. It introduces a synthetic-data training pipeline for a YOLOv5 detector to recognize custom components, and integrates a parametric assembly model with constraint-based optimization and structural analysis to dynamically update instructions in response to user actions. Demonstrations with LEGO and 3D-printed parts show substantial reductions in assembly time and high interaction accuracy, illustrating how deviations can become opportunities for exploration rather than errors. Overall, the work advances embodied making by tightly coupling perception, feedback, and adaptive guidance in AR to support creative, iterative assembly processes.

Abstract

Recent advances in augmented reality (AR) have enabled interactive systems that assist users in physical assembly tasks. In this paper, we present an AR-assisted assembly workflow that leverages object recognition and hand tracking to (1) identify custom components, (2) display step-by-step instructions, (3) detect assembly deviations, and (4) dynamically update the instructions based on users' hands-on interactions with physical parts. Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace. For each assembly step, it overlays bounding boxes in AR to indicate both the current position and the target placement of relevant components, while hand-tracking data verifies whether the user interacts with the correct part. Rather than enforcing a fixed sequence, the system highlights potential assembly errors and interprets user deviations as opportunities for iteration and creative exploration. A case study with LEGO blocks and custom 3D-printed components demonstrates how the system links digital instructions to physical assembly, eliminating the need for manual searching, sorting, or labeling of parts.
Paper Structure (9 sections, 10 figures)

This paper contains 9 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Object Recognition and Digital Twin of Physical Components in AR
  5. Step by Step Assembly Instruction Generation and Visualization in AR
  6. Hand Tracking for Detecting User Interactions, Errors and Creative Deviations
  7. Adaptive Instructions with Parametric Assembly Model and Structural Analysis
  8. Initial Results and Discussions
  9. GenAI Usage Disclosure

Figures (10)

  • Figure 1: System pipeline diagram illustrating the inputs, software components, and data flow
  • Figure 2: Automated synthetic data generation pipeline in Blender for training a deep learning-based object recognition model
  • Figure 3: Digital twin of the physical assembly space, generated using 2D video-based object recognition and projected into a 3D planar representation.
  • Figure 4: The interface guides users with step-by-step instructions, showing where to pick up and place each component.
  • Figure 5: Assembly instructions for a twist wall LEGO assembly, connecting physical components locations with digital instructions.
  • ...and 5 more figures