Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Merging Multiple Datasets for Improved Appearance-Based Gaze Estimation

Liang Wu, Bertram E. Shi

TL;DR

This work proposes two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module, which improve gaze estimation performance over the SOTA both individually and collectively.

Abstract

Multiple datasets have been created for training and testing appearance-based gaze estimators. Intuitively, more data should lead to better performance. However, combining datasets to train a single esti-mator rarely improves gaze estimation performance. One reason may be differences in the experimental protocols used to obtain the gaze sam-ples, resulting in differences in the distributions of head poses, gaze an-gles, illumination, etc. Another reason may be the inconsistency between methods used to define gaze angles (label mismatch). We propose two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module. Most state-of-the-art estimators merge information extracted from images of the two eyes and the entire face either in parallel or combine information from the eyes first then with the face. Our proposed Two-stage Transformer-based Gaze-feature Fusion (TTGF) method uses transformers to merge information from each eye and the face separately and then merge across the two eyes. We argue that this improves head pose invariance since changes in head pose affect left and right eye images in different ways. Our proposed Gaze Adaptation Module (GAM) method handles annotation inconsis-tency by applying a Gaze Adaption Module for each dataset to correct gaze estimates from a single shared estimator. This enables us to combine information across datasets despite differences in labeling. Our experi-ments show that these innovations improve gaze estimation performance over the SOTA both individually and collectively (by 10% - 20%). Our code is available at https://github.com/HKUST-NISL/GazeSetMerge.

Merging Multiple Datasets for Improved Appearance-Based Gaze Estimation

TL;DR

This work proposes two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module, which improve gaze estimation performance over the SOTA both individually and collectively.

Abstract

Multiple datasets have been created for training and testing appearance-based gaze estimators. Intuitively, more data should lead to better performance. However, combining datasets to train a single esti-mator rarely improves gaze estimation performance. One reason may be differences in the experimental protocols used to obtain the gaze sam-ples, resulting in differences in the distributions of head poses, gaze an-gles, illumination, etc. Another reason may be the inconsistency between methods used to define gaze angles (label mismatch). We propose two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module. Most state-of-the-art estimators merge information extracted from images of the two eyes and the entire face either in parallel or combine information from the eyes first then with the face. Our proposed Two-stage Transformer-based Gaze-feature Fusion (TTGF) method uses transformers to merge information from each eye and the face separately and then merge across the two eyes. We argue that this improves head pose invariance since changes in head pose affect left and right eye images in different ways. Our proposed Gaze Adaptation Module (GAM) method handles annotation inconsis-tency by applying a Gaze Adaption Module for each dataset to correct gaze estimates from a single shared estimator. This enables us to combine information across datasets despite differences in labeling. Our experi-ments show that these innovations improve gaze estimation performance over the SOTA both individually and collectively (by 10% - 20%). Our code is available at https://github.com/HKUST-NISL/GazeSetMerge.
Paper Structure (13 sections, 7 equations, 2 figures, 4 tables)

This paper contains 13 sections, 7 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Annotation Inconsistency
  4. Method
  5. Feature Fusion with Transformers
  6. Gaze Adaptation Module
  7. Architecture Details
  8. Experiments
  9. Comparison with state-of-the-art methods
  10. Ablation Study
  11. Effect of TTGF
  12. Effect of GAM
  13. Conclusion

Figures (2)

  • Figure 1: The proposed framework contains two modules: 1) TTGF and 2) GAM. The TTGF applies two-stage feature fusion to the features of the head and eyes with transformers, and the GAM produces a gaze offset to adjust the predicted gaze for mixed datasets training.
  • Figure 2: Four types of feature fusion for gaze estimation models: (a) two-eyes model uses the cropped eye patches as inputs. (b) PAR indicates left eye, right eye, and head features are combined in parallel. (c) LR-EH indicates that left and right eye features are combined first then combined head features. (d) EH-LR indicates that single eye and head features are combined first followed by a combination across the left and right..