GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer

TL;DR

GazeFormer-MoE tackles robust 3D gaze estimation under diverse illumination, pose, and background conditions by fusing CLIP-based semantic context with multi-scale visual tokens in a routed/shared Mixture of Experts Transformer. The method conditions global CLIP features with learnable prototype banks, unifies global, patch, and CNN tokens in a single attention space, and uses MoE routing to adapt capacity to appearance sub-distributions. On four benchmarks (MPIIFaceGaze, EYEDIAP, Gaze360, ETH-XGaze), it achieves new state-of-the-art angular errors with up to 64% relative improvement, and ablations show the combination of semantic conditioning, cross-scale fusion, and MoE as key drivers. Limitations include a discrete prototype vocabulary and single-frame processing, with future work aiming at dynamic prototypes and temporally aware sparse experts to further boost robustness and efficiency.

Abstract

We present a semantics modulated, multi scale Transformer for 3D gaze estimation. Our model conditions CLIP global features with learnable prototype banks (illumination, head pose, background, direction), fuses these prototype-enriched global vectors with CLIP patch tokens and high-resolution CNN tokens in a unified attention space, and replaces several FFN blocks with routed/shared Mixture of Experts to increase conditional capacity. Evaluated on MPIIFaceGaze, EYEDIAP, Gaze360 and ETH-XGaze, our model achieves new state of the art angular errors of 2.49°, 3.22°, 10.16°, and 1.44°, demonstrating up to a 64% relative improvement over previously reported results. ablations attribute gains to prototype conditioning, cross scale fusion, MoE and hyperparameter. Our code is publicly available at https://github. com/AIPMLab/Gazeformer.

Figures (1)

• Figure 1: (Left) CLIP Feature Extraction: A pre-trained CLIP text encoder processes four prompt categories (Illuminations, Backgrounds, Head Poses, Descriptions), which are fused with global image features via context selection to form semantic features $\mathbf{f}_1,\mathbf{f}_2$. Meanwhile, a CNN extracts local visual maps. (Right) Transformer Gaze Estimation: Multi-modal features ($\mathbf{f}1$, $\mathbf{f}2$, global patches $\mathbf{T}_{\text{patch}}$, CNN features $\mathbf{T}_{\text{cnn}}$) are projected, concatenated, and input into a Transformer encoder with MoE layers. The gating-based MoE enhances capacity and adaptability, yielding the final gaze prediction.