Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild

Jose Vargas Quiros, Chirag Raman, Stephanie Tan, Ekin Gedik, Laura Cabrera-Quiros, Hayley Hung

TL;DR

This paper revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event, and provides the signals and ground truth necessary to evaluate a wide range of speaking status detection methods.

Abstract

Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.

REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild

TL;DR

This paper revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event, and provides the signals and ground truth necessary to evaluate a wide range of speaking status detection methods.

Abstract

Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.
Paper Structure (20 sections, 4 figures, 2 tables)

This paper contains 20 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Related Datasets and Methods in Mingling Settings
  4. Speaking Status Detection in Non-mingling Settings
  5. Data Acquisition
  6. Participant Procedure
  7. Sensor setup
  8. Data Collection Details
  9. Data Annotation
  10. Automatic Audio-based Speaking Status Annotation
  11. Semi-automatic Pose Annotations
  12. Dataset Statistics
  13. Baselines for Automated Speaking Status Segmentation
  14. Evaluation Setup
  15. Video-based Speaking Status Segmentation
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Captures of our dataset and data subjects.
  • Figure 2: Seconds of speaking time per subject with speech data in REWIND dataset. Columns in green indicate subjects with complete information (audio, video, acceleration). Columns in yellow indicate subjects with audio and acceleration information, but who are not visible in the videos (no pose). Columns in red indicate subjects without any body movement information (only audio).
  • Figure 3: Distribution of length of contiguous speaking segments (s) in the speaking status detection labels for REWIND and MatchNMingle Cabrera-Quiros2020 datasets. REWIND shows greater temporal granularity (shorter segments), thanks to annotations having been obtained from audio.
  • Figure 4: Segmentation heads for acceleration and video models. The first block represents the feature map before the head of the ResNet model, for each modality method. Subsequent operations pool and convolve over the spatial and channel dimensions, and up-sample the time dimension.