OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild

TL;DR

OmniGaze tackles the generalization gap in 3D gaze estimation by leveraging large-scale unlabeled face data within a reward-driven semi-supervised pipeline. It trains a teacher on labeled data to generate pseudo labels for unlabeled images and employs a reward model that fuses geometry, visual embeddings via CLIP, and scene-specific gaze descriptions from Multimodal LLMs to assess pseudo-label reliability, producing a final confidence score for filtering and weighting. The student is trained with a supervised loss $L^{s}$ on labeled data and an unsupervised loss $L^{u}$ on high-quality pseudo-labeled data, with pseudo-labels refreshed periodically every $K$ epochs. Empirically, OmniGaze achieves state-of-the-art results on five datasets in-domain and cross-domain and demonstrates strong zero-shot generalization on unseen datasets, indicating its potential as a scalable automatic gaze annotation engine for real-world deployment, while highlighting considerations around data scale and computational costs.

Abstract

Current 3D gaze estimation methods struggle to generalize across diverse data domains, primarily due to i) the scarcity of annotated datasets, and ii) the insufficient diversity of labeled data. In this work, we present OmniGaze, a semi-supervised framework for 3D gaze estimation, which utilizes large-scale unlabeled data collected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse collection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning a broader distribution, OmniGaze adopts a standard pseudo-labeling strategy and devises a reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by an off-the-shelf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized to select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OmniGaze achieves state-of-the-art performance on five datasets under both in-domain and cross-domain settings. Furthermore, we also evaluate the efficacy of OmniGaze as a scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.

Figures (7)

• Figure 1: Left: By making efficient use of large-scale diverse unlabeled datasets via reward-driven pseudo label selection, our OminGaze can estimate high-quality 3D gaze directions for in-the-wild images in diverse conditions, e.g., extreme head poses, varying lighting conditions, and appearance, etc. Red and yellow arrows represent predictions from OmniGaze and base model. Right: our OminGaze achieves state-of-the-art performance on five datasets under three settings, i.e., in-domain, cross-domain, and zero-shot generalization.
• Figure 2: Overview of the proposed semi-supervised learning framework. (a) OminGaze jointly trains on both labeled data and large-scale unlabeled data, and utilizes a reward model to select and reweight high-quality pseudo labels for unlabeled data. (b) The reward model evaluates the reliability of pseudo labels by jointly reasoning over visual appearance, scene-specific gaze descriptions, and geometric gaze directions.
• Figure 3: Visual comparison results (§\ref{['sec::zeroshot']}) on four unseen datasets (left) and in the wild (right). Red and yellow arrows represent gaze estimation predictions from our OmniGaze and base model trained only on labeled datasets. Four datasets from left to right: MPIIFaceGaze mpiiface, EyeDiap eyediap, RT-Gene fischer2018rt, and IVGaze cheng2024you.
• Figure 4: Additional examples of each dataset(§\ref{['supp::diver']}). Our curated training dataset (i.e., labeled datasets and large-scale unlabeled data) exhibits wide variability in terms of capturing environments (e.g., indoor, outdoor, synthetic, or in-vehicle scenes), facial appearance, lighting conditions, head poses, etc.
• Figure 5: The impact of unlabeled data size (§\ref{['sec::quan']}).