Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation

Pengwei Yin, Jingjing Wang, Guanzhong Zeng, Di Xie, Jiang Zhu

TL;DR

This paper proposes a novel approach to gaze estimation, reframing the gaze estimation task as a vision-language alignment issue, and proposes a geometry-aware interpolation method to obtain more precise gaze embeddings.

Abstract

The ability of gaze estimation models to generalize is often significantly hindered by various factors unrelated to gaze, especially when the training dataset is limited. Current strategies aim to address this challenge through different domain generalization techniques, yet they have had limited success due to the risk of overfitting when solely relying on value labels for regression. Recent progress in pre-trained vision-language models has motivated us to capitalize on the abundant semantic information available. We propose a novel approach in this paper, reframing the gaze estimation task as a vision-language alignment issue. Our proposed framework, named Language-Guided Gaze Estimation (LG-Gaze), learns continuous and geometry-sensitive features for gaze estimation benefit from the rich prior knowledges of vision-language models. Specifically, LG-Gaze aligns gaze features with continuous linguistic features through our proposed multimodal contrastive regression loss, which customizes adaptive weights for different negative samples. Furthermore, to better adapt to the labels for gaze estimation task, we propose a geometry-aware interpolation method to obtain more precise gaze embeddings. Through extensive experiments, we validate the efficacy of our framework in four different cross-domain evaluation tasks.

LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation

TL;DR

This paper proposes a novel approach to gaze estimation, reframing the gaze estimation task as a vision-language alignment issue, and proposes a geometry-aware interpolation method to obtain more precise gaze embeddings.

Abstract

The ability of gaze estimation models to generalize is often significantly hindered by various factors unrelated to gaze, especially when the training dataset is limited. Current strategies aim to address this challenge through different domain generalization techniques, yet they have had limited success due to the risk of overfitting when solely relying on value labels for regression. Recent progress in pre-trained vision-language models has motivated us to capitalize on the abundant semantic information available. We propose a novel approach in this paper, reframing the gaze estimation task as a vision-language alignment issue. Our proposed framework, named Language-Guided Gaze Estimation (LG-Gaze), learns continuous and geometry-sensitive features for gaze estimation benefit from the rich prior knowledges of vision-language models. Specifically, LG-Gaze aligns gaze features with continuous linguistic features through our proposed multimodal contrastive regression loss, which customizes adaptive weights for different negative samples. Furthermore, to better adapt to the labels for gaze estimation task, we propose a geometry-aware interpolation method to obtain more precise gaze embeddings. Through extensive experiments, we validate the efficacy of our framework in four different cross-domain evaluation tasks.

Paper Structure

This paper contains 27 sections, 9 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Gaze Estimation
  4. Vision Language Model
  5. Contrastive Learning
  6. Method
  7. Problem Statement
  8. LG-Gaze Framework
  9. Geometry-aware Continuous Gaze Prompts
  10. Continuous Multimodal Contrastive Regression Loss Function
  11. Overall Objective
  12. Experiments
  13. Experiment Details
  14. Data Preparation
  15. Vision Model Implementation Details
  16. ...and 12 more sections

Figures (4)

  • Figure 1: (a) The traditional method of gaze generalization involves overseeing model training by means of numerical label regression. (b) In our study, we introduce a text models to steer the development of robust features in visual models.
  • Figure 2: We reformulate the task as an image-language matching problem, which mainly consists of a trainable prompt, a frozen text encoder, a trainable image encoder.
  • Figure 3: These four subfigures contain all steps of our proposed interpolation method.
  • Figure 4: Illustration of the feature distribution. Various colors indicate different gaze directions and similar gaze directions have similar colors. (Recommend viewing in color).