Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The VVAD-LRS3 Dataset for Visual Voice Activity Detection

Adrian Lubitz, Matias Valdenegro-Toro, Frank Kirchner

TL;DR

The VVAD-LRS3 paper tackles the scarcity of Visual Voice Activity Detection data by creating a large, LRS3-derived dataset for VVAD. It builds a flexible pipeline to convert TED talk videos into speaking/not-speaking samples, producing four feature flavors (face images, lip images, and their landmark features) and yielding 44,489 samples (train/validation/test). Baseline and end-to-end CNN–LSTM models show strong performance, with the best end-to-end face-image models reaching about 92% test accuracy, exceeding human performance on the same task. The dataset, metadata schema, and baselines are publicly available to accelerate robust VVAD development for social robots and related applications.

Abstract

Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92% was reached on the test set. A study with humans showed that they reach an accuracy of 87.93% on the test set.

The VVAD-LRS3 Dataset for Visual Voice Activity Detection

TL;DR

The VVAD-LRS3 paper tackles the scarcity of Visual Voice Activity Detection data by creating a large, LRS3-derived dataset for VVAD. It builds a flexible pipeline to convert TED talk videos into speaking/not-speaking samples, producing four feature flavors (face images, lip images, and their landmark features) and yielding 44,489 samples (train/validation/test). Baseline and end-to-end CNN–LSTM models show strong performance, with the best end-to-end face-image models reaching about 92% test accuracy, exceeding human performance on the same task. The dataset, metadata schema, and baselines are publicly available to accelerate robust VVAD development for social robots and related applications.

Abstract

Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92% was reached on the test set. A study with humans showed that they reach an accuracy of 87.93% on the test set.

Paper Structure

This paper contains 72 sections, 7 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. DATASET CAPTURE
  4. Text
  5. Ref
  6. FRAME
  7. WORD
  8. Face Images
  9. Lip Images
  10. Face Features
  11. Lip Features
  12. INITIAL EXPERIMENTAL RESULTS
  13. CONCLUSIONS AND FUTURE WORK
  14. Appendix
  15. Datasheet for VVAD-LRS3 Dataset
  16. ...and 57 more sections

Figures (7)

  • Figure 1: Example of error detection - Person is classified as having a mouth activity, however does not speak Bendris2010
  • Figure 2: Visualization of one frame of different features.
  • Figure 3: Random selection of speaking (positive) and negative (not speaking) samples from the VVAD-LRS3 dataset
  • Figure 4: Comparison of performance as image size and number of timesteps/frames is varied on MobileNet.
  • Figure 5: Sample 6178 is labeled as a negative (not speaking) sample by the automatic transformation from LRS3 to VVAD dataset. On the human accuracy level test 100% of the subjects classified the sample as positive (speaking) sample. Beat boxing is not considered speech in the LRS3 dataset, which causes the wrong label.
  • ...and 2 more figures