Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Assembly Progress Estimation via Deep Metric Learning

Kazuma Miura, Sarthak Pathak, Kazunori Umeda

TL;DR

This work tackles robust assembly progress estimation in settings where manual, multi-day tasks produce only subtle visual changes and occlusions. It introduces Anomaly Quadruplet-Net, a deep metric-learning framework that extends Anomaly Triplet-Net with Quadruplet Loss, a redesigned data loader, pseudo-occlusion via Random Erasing, and a YOLOv8-based detector for cropping product images, combined with kNN-based inference in the learned feature space. On a small desktop PC assembly dataset, AQN achieves 97.8% test accuracy and 2.9% misclassification between adjacent steps, improving over the prior method by 1.3 percentage points and reducing adjacent-step errors by 1.9 points. The approach demonstrates occlusion-robust progress estimation using fixed cameras and limited data, with plans to extend to video-based estimation and marker-assisted tracking in real factories.

Abstract

In recent years, the advancement of AI technologies has accelerated the development of smart factories. In particular, the automatic monitoring of product assembly progress is crucial for improving operational efficiency, minimizing the cost of discarded parts, and maximizing factory productivity. However, in cases where assembly tasks are performed manually over multiple days, implementing smart factory systems remains a challenge. Previous work has proposed Anomaly Triplet-Net, which estimates assembly progress by applying deep metric learning to the visual features of products. Nevertheless, when visual changes between consecutive tasks are subtle, misclassification often occurs. To address this issue, this paper proposes a robust system for estimating assembly progress, even in cases of occlusion or minimal visual change, using a small-scale dataset. Our method leverages a Quadruplet Loss-based learning approach for anomaly images and introduces a custom data loader that strategically selects training samples to enhance estimation accuracy. We evaluated our approach using a image datasets: captured during desktop PC assembly. The proposed Anomaly Quadruplet-Net outperformed existing methods on the dataset. Specifically, it improved the estimation accuracy by 1.3% and reduced misclassification between adjacent tasks by 1.9% in the desktop PC dataset and demonstrating the effectiveness of the proposed method.

Robust Assembly Progress Estimation via Deep Metric Learning

TL;DR

This work tackles robust assembly progress estimation in settings where manual, multi-day tasks produce only subtle visual changes and occlusions. It introduces Anomaly Quadruplet-Net, a deep metric-learning framework that extends Anomaly Triplet-Net with Quadruplet Loss, a redesigned data loader, pseudo-occlusion via Random Erasing, and a YOLOv8-based detector for cropping product images, combined with kNN-based inference in the learned feature space. On a small desktop PC assembly dataset, AQN achieves 97.8% test accuracy and 2.9% misclassification between adjacent steps, improving over the prior method by 1.3 percentage points and reducing adjacent-step errors by 1.9 points. The approach demonstrates occlusion-robust progress estimation using fixed cameras and limited data, with plans to extend to video-based estimation and marker-assisted tracking in real factories.

Abstract

In recent years, the advancement of AI technologies has accelerated the development of smart factories. In particular, the automatic monitoring of product assembly progress is crucial for improving operational efficiency, minimizing the cost of discarded parts, and maximizing factory productivity. However, in cases where assembly tasks are performed manually over multiple days, implementing smart factory systems remains a challenge. Previous work has proposed Anomaly Triplet-Net, which estimates assembly progress by applying deep metric learning to the visual features of products. Nevertheless, when visual changes between consecutive tasks are subtle, misclassification often occurs. To address this issue, this paper proposes a robust system for estimating assembly progress, even in cases of occlusion or minimal visual change, using a small-scale dataset. Our method leverages a Quadruplet Loss-based learning approach for anomaly images and introduces a custom data loader that strategically selects training samples to enhance estimation accuracy. We evaluated our approach using a image datasets: captured during desktop PC assembly. The proposed Anomaly Quadruplet-Net outperformed existing methods on the dataset. Specifically, it improved the estimation accuracy by 1.3% and reduced misclassification between adjacent tasks by 1.9% in the desktop PC dataset and demonstrating the effectiveness of the proposed method.
Paper Structure (20 sections, 3 equations, 49 figures)

This paper contains 20 sections, 3 equations, 49 figures.

Table of Contents

  1. Introduction
  2. Research Background
  3. Related Work
  4. Approaches Based on Skeletal Keypoint Information
  5. Approaches Focusing on Product State
  6. Approaches Focusing on Product Components
  7. Research Objective
  8. Proposedmethod
  9. Assembly Progress Estimation Method
  10. Assembly product etection
  11. Anomaly Quadruplet-net:modelsthatconsider occlusion
  12. Overall Structure of Anomaly Quadruplet-Net
  13. Training Phase
  14. Inference Phase
  15. Step Estimation Using the k-Nearest Neighbor Method
  16. ...and 5 more sections

Figures (49)

  • Figure 1: SCARA robot used in an assembly line robo1
  • Figure 2: Large-scale welding robot used in an assembly line robo2
  • Figure 3: Overview of a human--robot collaborative cell production system cell1
  • Figure 4: Example of actual collaborative work between a human and a robot cell1
  • Figure 5: Progress estimation based on product state toshiba
  • ...and 44 more figures