Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A vision-based framework for human behavior understanding in industrial assembly lines

Konstantinos Papoutsakis, Nikolaos Bakalos, Konstantinos Fragkoulis, Athena Zacharia, Georgia Kapetadimitri, Maria Pateraki

TL;DR

Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.

Abstract

This paper introduces a vision-based framework for capturing and understanding human behavior in industrial assembly lines, focusing on car door manufacturing. The framework leverages advanced computer vision techniques to estimate workers' locations and 3D poses and analyze work postures, actions, and task progress. A key contribution is the introduction of the CarDA dataset, which contains domain-relevant assembly actions captured in a realistic setting to support the analysis of the framework for human pose and action analysis. The dataset comprises time-synchronized multi-camera RGB-D videos, motion capture data recorded in a real car manufacturing environment, and annotations for EAWS-based ergonomic risk scores and assembly activities. Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.

A vision-based framework for human behavior understanding in industrial assembly lines

TL;DR

Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.

Abstract

This paper introduces a vision-based framework for capturing and understanding human behavior in industrial assembly lines, focusing on car door manufacturing. The framework leverages advanced computer vision techniques to estimate workers' locations and 3D poses and analyze work postures, actions, and task progress. A key contribution is the introduction of the CarDA dataset, which contains domain-relevant assembly actions captured in a realistic setting to support the analysis of the framework for human pose and action analysis. The dataset comprises time-synchronized multi-camera RGB-D videos, motion capture data recorded in a real car manufacturing environment, and annotations for EAWS-based ergonomic risk scores and assembly activities. Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.
Paper Structure (24 sections, 5 figures, 5 tables)

This paper contains 24 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Human pose estimation
  4. Vision-based ergonomic posture evaluation
  5. Human actions monitoring
  6. Datasets
  7. CarDA - Car door Assembly Activities Dataset
  8. Deployment environment
  9. Car Door Assembly Activities
  10. Data Acquisition
  11. Data Annotation
  12. Ground truth data on human localization and 3D pose estimation:
  13. Ground truth data on basic postures of the EAWS ergonomic screening tool:
  14. Ground truth related to temporal segmentation and classification of car door assembly actions (subgoals) during task cycle executions:
  15. Human behavior understanding framework
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Workflow of the human behavior understanding framework.
  • Figure 2: The shopfloor outline of the real manufacturing environment used to acquire the CarDA dataset. Three workstation areas (color-coded rectangles) are virtually defined on the conveyor belt area (assembly line). A pair of two stereo cameras (InXX, OutXX) (blue cylinders) are installed on both sides of each workstation WSXX.
  • Figure 3: Annotations of the car door and humans area (marked in yellow). Annotations of the car door with (left) and without (middle) the overlapping human area and the extracted 2D segmentation masks of the humans (right).
  • Figure 4: (a) Experimental results are illustrated as a confusion matrix for (a) the vision-based classification of EAWS-based ergonomic postures (TR: trunk rotation, LB: lateral bend) as shown in Sec. \ref{['sec:ergonomics']}, (b) the progress monitoring of car door assembly activities during task cycles using the proposed Car Door Assembly dataset.
  • Figure 5: Performance evaluation of the human action monitoring (a) Macro-Average Receiver Operating Characteristic curve, (b) Recall, F1, AUC scores (validation set).