Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Person Identification from Egocentric Human-Object Interactions using 3D Hand Pose

Muhammad Hamza, Danish Hamid, Muhammad Tahir Akram

TL;DR

This paper tackles user identification from egocentric hand-object interactions by introducing I2S (Interact2Sign), a multi-stage framework that relies on handcrafted 3D hand pose features to perform object detection, HOI recognition, and subject identification. A novel IHSE descriptor is developed to capture inter-hand coordination in bimanual interactions, and the system is trained with an augmented ARCTIC+H2O dataset using XGBoost in a three-stage pipeline. The approach achieves high performance, with an overall pipeline F1-score around 97.5% and near-real-time on-device inference while keeping model size under 4 MB, showcasing a lightweight, interpretable alternative to deep models for edge AR authentication. This work demonstrates the viability of privacy-preserving, on-device user identification through HOI patterns and highlights future opportunities in hybridizing handcrafted descriptors with deep representations and expanding interaction vocabularies.

Abstract

Human-Object Interaction Recognition (HOIR) and user identification play a crucial role in advancing augmented reality (AR)-based personalized assistive technologies. These systems are increasingly being deployed in high-stakes, human-centric environments such as aircraft cockpits, aerospace maintenance, and surgical procedures. This research introduces I2S (Interact2Sign), a multi stage framework designed for unobtrusive user identification through human object interaction recognition, leveraging 3D hand pose analysis in egocentric videos. I2S utilizes handcrafted features extracted from 3D hand poses and per forms sequential feature augmentation: first identifying the object class, followed by HOI recognition, and ultimately, user identification. A comprehensive feature extraction and description process was carried out for 3D hand poses, organizing the extracted features into semantically meaningful categories: Spatial, Frequency, Kinematic, Orientation, and a novel descriptor introduced in this work, the Inter-Hand Spatial Envelope (IHSE). Extensive ablation studies were conducted to determine the most effective combination of features. The optimal configuration achieved an impressive average F1-score of 97.52% for user identification, evaluated on a bimanual object manipulation dataset derived from the ARCTIC and H2O datasets. I2S demonstrates state-of-the-art performance while maintaining a lightweight model size of under 4 MB and a fast inference time of 0.1 seconds. These characteristics make the proposed framework highly suitable for real-time, on-device authentication in security-critical, AR-based systems.

Person Identification from Egocentric Human-Object Interactions using 3D Hand Pose

TL;DR

This paper tackles user identification from egocentric hand-object interactions by introducing I2S (Interact2Sign), a multi-stage framework that relies on handcrafted 3D hand pose features to perform object detection, HOI recognition, and subject identification. A novel IHSE descriptor is developed to capture inter-hand coordination in bimanual interactions, and the system is trained with an augmented ARCTIC+H2O dataset using XGBoost in a three-stage pipeline. The approach achieves high performance, with an overall pipeline F1-score around 97.5% and near-real-time on-device inference while keeping model size under 4 MB, showcasing a lightweight, interpretable alternative to deep models for edge AR authentication. This work demonstrates the viability of privacy-preserving, on-device user identification through HOI patterns and highlights future opportunities in hybridizing handcrafted descriptors with deep representations and expanding interaction vocabularies.

Abstract

Human-Object Interaction Recognition (HOIR) and user identification play a crucial role in advancing augmented reality (AR)-based personalized assistive technologies. These systems are increasingly being deployed in high-stakes, human-centric environments such as aircraft cockpits, aerospace maintenance, and surgical procedures. This research introduces I2S (Interact2Sign), a multi stage framework designed for unobtrusive user identification through human object interaction recognition, leveraging 3D hand pose analysis in egocentric videos. I2S utilizes handcrafted features extracted from 3D hand poses and per forms sequential feature augmentation: first identifying the object class, followed by HOI recognition, and ultimately, user identification. A comprehensive feature extraction and description process was carried out for 3D hand poses, organizing the extracted features into semantically meaningful categories: Spatial, Frequency, Kinematic, Orientation, and a novel descriptor introduced in this work, the Inter-Hand Spatial Envelope (IHSE). Extensive ablation studies were conducted to determine the most effective combination of features. The optimal configuration achieved an impressive average F1-score of 97.52% for user identification, evaluated on a bimanual object manipulation dataset derived from the ARCTIC and H2O datasets. I2S demonstrates state-of-the-art performance while maintaining a lightweight model size of under 4 MB and a fast inference time of 0.1 seconds. These characteristics make the proposed framework highly suitable for real-time, on-device authentication in security-critical, AR-based systems.

Paper Structure

This paper contains 30 sections, 7 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Biometrics
  4. Egocentric Datasets
  5. Human-Object Interaction (HOI) Recognition in First-Person View
  6. Egocentric User-Identification
  7. Hand-Based Feature Extraction
  8. Methodology
  9. Data Acquisition and Preparation
  10. Feature Extraction and Description
  11. Spatial Feature Descriptor
  12. Orientation Feature Descriptor
  13. Kinematic Feature Descriptor
  14. Frequency-Domain Features
  15. Inter-Hand Spatial Envelope (IHSE)
  16. ...and 15 more sections

Figures (8)

  • Figure 1: Augmented Reality-assisted surgeryandersen2017augmented(left) and mechanical assembly(right)
  • Figure 2: Comparison of hand-pose variation in egocentric datasets Fan2022.
  • Figure 3: I2S: Multi-Stage Framework for HOIR-based User Identification
  • Figure 4: ARCTIC Objects: Each line depicts the axis of articulation Fan2022
  • Figure 5: Feature Extraction and Description Pipeline
  • ...and 3 more figures