Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data

Christian Stippel, Thomas Heitzinger, Rafael Sterzinger, Martin Kampel

TL;DR

The paper tackles privacy and lighting limitations in human behavior analysis by bridging RGB with depth and thermal modalities through a background-conditioned RGB→depth/thermal translation pipeline. It leverages ImageBind for background matching, YOLOv6 and SAM for robust human masks, and a Pix2Pix-based translator conditioned on background and signed distance fields, followed by post-processing to integrate subjects into new modalities. Evaluations show that integrating background, SDF, and cropped RGB improves perceptual and semantic alignment (FID/KID/MSE), and that synthetic data, when combined with real data, can match real-data performance in action recognition. This approach offers a scalable, privacy-preserving path to train HBA models in challenging conditions and provides a modular framework amenable to future diffusion or transformer-based enhancements.

Abstract

In pervasive machine learning, especially in Human Behavior Analysis (HBA), RGB has been the primary modality due to its accessibility and richness of information. However, linked with its benefits are challenges, including sensitivity to lighting conditions and privacy concerns. One possibility to overcome these vulnerabilities is to resort to different modalities. For instance, thermal is particularly adept at accentuating human forms, while depth adds crucial contextual layers. Despite their known benefits, only a few HBA-specific datasets that integrate these modalities exist. To address this shortage, our research introduces a novel generative technique for creating trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique capitalizes on human segmentation masks derived from RGB images, combined with thermal and depth backgrounds that are sourced automatically. With these two ingredients, we synthesize depth and thermal counterparts from existing RGB data utilizing conditional image-to-image translation. By employing this approach, we generate trimodal data that can be leveraged to train models for settings with limited data, bad lightning conditions, or privacy-sensitive areas.

Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data

TL;DR

The paper tackles privacy and lighting limitations in human behavior analysis by bridging RGB with depth and thermal modalities through a background-conditioned RGB→depth/thermal translation pipeline. It leverages ImageBind for background matching, YOLOv6 and SAM for robust human masks, and a Pix2Pix-based translator conditioned on background and signed distance fields, followed by post-processing to integrate subjects into new modalities. Evaluations show that integrating background, SDF, and cropped RGB improves perceptual and semantic alignment (FID/KID/MSE), and that synthetic data, when combined with real data, can match real-data performance in action recognition. This approach offers a scalable, privacy-preserving path to train HBA models in challenging conditions and provides a modular framework amenable to future diffusion or transformer-based enhancements.

Abstract

In pervasive machine learning, especially in Human Behavior Analysis (HBA), RGB has been the primary modality due to its accessibility and richness of information. However, linked with its benefits are challenges, including sensitivity to lighting conditions and privacy concerns. One possibility to overcome these vulnerabilities is to resort to different modalities. For instance, thermal is particularly adept at accentuating human forms, while depth adds crucial contextual layers. Despite their known benefits, only a few HBA-specific datasets that integrate these modalities exist. To address this shortage, our research introduces a novel generative technique for creating trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique capitalizes on human segmentation masks derived from RGB images, combined with thermal and depth backgrounds that are sourced automatically. With these two ingredients, we synthesize depth and thermal counterparts from existing RGB data utilizing conditional image-to-image translation. By employing this approach, we generate trimodal data that can be leveraged to train models for settings with limited data, bad lightning conditions, or privacy-sensitive areas.
Paper Structure (12 sections, 5 equations, 4 figures, 4 tables)

This paper contains 12 sections, 5 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Locate Background
  5. Obtain Human Masks
  6. Pre-Process Data
  7. Translate Data
  8. Post-Process Outputs
  9. Evaluation & Results
  10. Translation Performance
  11. Action Recognition Performance
  12. Conclusion

Figures (4)

  • Figure 1: Comparison of RGB, Thermal, and Depth Data
  • Figure 2: Illustration of the RGB to depth and thermal data transformation quality.
  • Figure 3: Overview of our proposed methodology, illustrating the integration of ImageBind for obtaining matching backgrounds, YOLOv6 and Segment Anything Model for segmenting human masks from RGB, and Pix2Pix for modality translation.
  • Figure 4: Visualization of extracting RGB and the normalized Signed Distance Field.