Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection

Roberto Daza, Aythami Morales, Julian Fierrez, Ruben Tolosana, Ruben Vera-Rodriguez

TL;DR

This work introduces mEBAL2, the largest public eyeblink detection database, collecting RGB and two NIR streams plus EEG from 180 students across MOOC-based tasks to study attention and cognitive load. It proposes three data-driven detectors, including frame-level OE-ConvNet, late-fusion LI-RI-RGB-ConvNet, and video-level OE-ConvLSTM, and validates them in frame- and sequence-level benchmarks. The results show RGB generally outperforms NIR at frame level, but combining spectra via multispectral training yields modest gains and improves generalization, with ConvLSTM achieving around $0.99$ accuracy on video sequences. Generalization to the wild HUST-LEBW dataset confirms the dataset's value for training robust eyeblink detectors, highlighting mEBAL2 as a resource for advancing data-driven eyeblink detection and related applications like attention estimation and presentation attack detection. Future work includes deeper exploration of the NIR spectrum and temporal architectures to further leverage multispectral and sequential information.

Abstract

This work introduces a new multispectral database and novel approaches for eyeblink detection in RGB and Near-Infrared (NIR) individual images. Our contributed dataset (mEBAL2, multimodal Eye Blink and Attention Level estimation, Version 2) is the largest existing eyeblink database, representing a great opportunity to improve data-driven multispectral approaches for blink detection and related applications (e.g., attention level estimation and presentation attack detection in face biometrics). mEBAL2 includes 21,100 image sequences from 180 different students (more than 2 million labeled images in total) while conducting a number of e-learning tasks of varying difficulty or taking a real course on HTML initiation through the edX MOOC platform. mEBAL2 uses multiple sensors, including two Near-Infrared (NIR) and one RGB camera to capture facial gestures during the execution of the tasks, as well as an Electroencephalogram (EEG) band to get the cognitive activity of the user and blinking events. Furthermore, this work proposes a Convolutional Neural Network architecture as benchmark for blink detection on mEBAL2 with performances up to 97%. Different training methodologies are implemented using the RGB spectrum, NIR spectrum, and the combination of both to enhance the performance on existing eyeblink detectors. We demonstrate that combining NIR and RGB images during training improves the performance of RGB eyeblink detectors (i.e., detection based only on a RGB image). Finally, the generalization capacity of the proposed eyeblink detectors is validated in wilder and more challenging environments like the HUST-LEBW dataset to show the usefulness of mEBAL2 to train a new generation of data-driven approaches for eyeblink detection.

mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection

TL;DR

This work introduces mEBAL2, the largest public eyeblink detection database, collecting RGB and two NIR streams plus EEG from 180 students across MOOC-based tasks to study attention and cognitive load. It proposes three data-driven detectors, including frame-level OE-ConvNet, late-fusion LI-RI-RGB-ConvNet, and video-level OE-ConvLSTM, and validates them in frame- and sequence-level benchmarks. The results show RGB generally outperforms NIR at frame level, but combining spectra via multispectral training yields modest gains and improves generalization, with ConvLSTM achieving around accuracy on video sequences. Generalization to the wild HUST-LEBW dataset confirms the dataset's value for training robust eyeblink detectors, highlighting mEBAL2 as a resource for advancing data-driven eyeblink detection and related applications like attention estimation and presentation attack detection. Future work includes deeper exploration of the NIR spectrum and temporal architectures to further leverage multispectral and sequential information.

Abstract

This work introduces a new multispectral database and novel approaches for eyeblink detection in RGB and Near-Infrared (NIR) individual images. Our contributed dataset (mEBAL2, multimodal Eye Blink and Attention Level estimation, Version 2) is the largest existing eyeblink database, representing a great opportunity to improve data-driven multispectral approaches for blink detection and related applications (e.g., attention level estimation and presentation attack detection in face biometrics). mEBAL2 includes 21,100 image sequences from 180 different students (more than 2 million labeled images in total) while conducting a number of e-learning tasks of varying difficulty or taking a real course on HTML initiation through the edX MOOC platform. mEBAL2 uses multiple sensors, including two Near-Infrared (NIR) and one RGB camera to capture facial gestures during the execution of the tasks, as well as an Electroencephalogram (EEG) band to get the cognitive activity of the user and blinking events. Furthermore, this work proposes a Convolutional Neural Network architecture as benchmark for blink detection on mEBAL2 with performances up to 97%. Different training methodologies are implemented using the RGB spectrum, NIR spectrum, and the combination of both to enhance the performance on existing eyeblink detectors. We demonstrate that combining NIR and RGB images during training improves the performance of RGB eyeblink detectors (i.e., detection based only on a RGB image). Finally, the generalization capacity of the proposed eyeblink detectors is validated in wilder and more challenging environments like the HUST-LEBW dataset to show the usefulness of mEBAL2 to train a new generation of data-driven approaches for eyeblink detection.
Paper Structure (17 sections, 2 figures, 5 tables)

This paper contains 17 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Eyeblink detection databases
  4. Eyeblink detection methods
  5. mEBAL2 Database
  6. Tasks
  7. Proposed Eyeblink Detector
  8. ROI detection
  9. Eyeblink verification architecture
  10. Experiments and Results
  11. Experimental protocol
  12. Experiments: mEBAL2 Benchmark
  13. mEBAL2: Blink detection at frame-level
  14. mEBAL2: Blink detection at video-sequence level
  15. Experiments on HUST-LEBW: Evaluating the generalization ability of models trained with mEBAL2
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Different examples from mEBAL2. (top) Sequence images with variations in illumination, posing, and distance to the camera. (bottom) Examples of eyeblink and no-blink with RGB and NIR images.
  • Figure 2: F1 score results on HUST-LEBW evaluation for different training ratios in mEBAL2 for OE-ConvNet and Soukupova soukupova2016eye + Insightface architectures.