GlovEgo-HOI: Bridging the Synthetic-to-Real Gap for Industrial Egocentric Human-Object Interaction Detection

TL;DR

GlovEgo-HOI tackles the data scarcity challenge in industrial EHOI by blending synthetic data generation with diffusion-based glove augmentation to create a PPE-aware benchmark. It introduces GlovEgo-Net, a multimodal architecture that enriches hand-object interaction detection with a Glove Head and a Keypoint Head, leveraging 21 hand keypoints and PPE cues. Extensive experiments show that pre-training on synthetic data and fine-tuning on real data yields substantial improvements across EHOI metrics and enables better zero-shot transfer, outperforming Real-Only baselines. The work provides an open-source dataset, augmentation pipeline, and pretrained models to accelerate industrial EHOI research and practical deployment.

Abstract

Egocentric Human-Object Interaction (EHOI) analysis is crucial for industrial safety, yet the development of robust models is hindered by the scarcity of annotated domain-specific data. We address this challenge by introducing a data generation framework that combines synthetic data with a diffusion-based process to augment real-world images with realistic Personal Protective Equipment (PPE). We present GlovEgo-HOI, a new benchmark dataset for industrial EHOI, and GlovEgo-Net, a model integrating Glove-Head and Keypoint- Head modules to leverage hand pose information for enhanced interaction detection. Extensive experiments demonstrate the effectiveness of the proposed data generation framework and GlovEgo-Net. To foster further research, we release the GlovEgo-HOI dataset, augmentation pipeline, and pre-trained models at: GitHub project.

Figures (7)

• Figure 1: (a) GlovEgo-HOI Dataset: Examples from our proposed GlovEgo-HOI dataset. The top row shows synthetic images, while the bottom row contains real-world images augmented with PPE via our diffusion-model-based pipeline. (b) GlovEgo-Net Training Strategies: Comparison of GlovEgo-Net performance under different training regimes. (Top) Results with real-world data only (Real-Only). (Bottom) Results with our Synth+Real approach, using synthetic data for pre-training and real samples for fine-tuning, superior contact state understanding. Our results show that while Real-Only models saturate quickly, the inclusion of synthetic keypoints enables GlovEgo-Net to accurately detect interactions even in complex industrial scenes.
• Figure 2: GlovEgo-HOI-Synth samples featuring multimodal annotations: RGB, depth, masks, hand keypoints, and glove masks.
• Figure 3: Diffusion artifacts (e.g., background hallucinations) identified and removed via our SSIM-based validation pipeline.
• Figure 4: Augmentation pipeline for GlovEgo-HOI-Real: original frames (top) and corresponding FLUX-augmented outputs (bottom).
• Figure 5: Overview of the GlovEgo-Net architecture. The system processes RGB images to extract multimodal features, fusing pose heatmaps, depth, and masks through Early Fusion (EF) and Late Fusion (LF) to output final EHOI quadruples.