TextGaze: Gaze-Controllable Face Generation with Natural Language

TL;DR

The paper tackles natural language-driven gaze control for face generation by introducing ToG, a text-of-gaze dataset with over 95k annotations that describe head pose and gaze behavior. It presents a two-stage gaze-controllable text-to-face framework: first, a text-to-gaze diffusion model with a Text Attention Module maps text to pose $ig\{\boldsymbol{g}, \boldsymbol{h}\big\}$, then a 3D-face-model prior (FLAME) yields a 2D face sketch used by a ControlNet-conditioned diffusion model to render the final image. This approach avoids explicit gaze labels while achieving accurate head/gaze alignment with textual descriptions, and it includes extensive experiments on FFHQ showing superior or competitive performance against baselines, along with a dataset and code release. The contribution advances practical, natural-language specification of gaze in facial synthesis, with potential impact on visual realism, human-computer interaction, and digital human applications.

Abstract

Generating face image with specific gaze information has attracted considerable attention. Existing approaches typically input gaze values directly for face generation, which is unnatural and requires annotated gaze datasets for training, thereby limiting its application. In this paper, we present a novel gaze-controllable face generation task. Our approach inputs textual descriptions that describe human gaze and head behavior and generates corresponding face images. Our work first introduces a text-of-gaze dataset containing over 90k text descriptions spanning a dense distribution of gaze and head poses. We further propose a gaze-controllable text-to-face method. Our method contains a sketch-conditioned face diffusion module and a model-based sketch diffusion module. We define a face sketch based on facial landmarks and eye segmentation map. The face diffusion module generates face images from the face sketch, and the sketch diffusion module employs a 3D face model to generate face sketch from text description. Experiments on the FFHQ dataset show the effectiveness of our method. We will release our dataset and code for future research.

Figures (4)

• Figure 1: Visualization of head and gaze scores of different subjects.
• Figure 2: The model operates in two distinct stages: pose generation and face generation. During the first stage, the text description is initially fed into the CLIP model to obtain word embeddings. These embeddings are then processed through our TAM module as the condition to gaze diffusion module. The gaze diffusion module generates head pose and gaze direction matched with input text. Subsequently, these poses are utilized to rotate a 3D face model into the predicted orientation. The rotated model is then projected into a two-dimensional space, resulting in the creation of a sketch. In the second stage, the face image is meticulously crafted using a diffusion model, which is specifically conditioned on the sketches generated in the previous stage. This two-tiered approach ensures a coherent and detailed synthesis of facial images.
• Figure 3: Visulization of generated images from our model and two baselines esser2021tamingzhang2023adding. We show 12 sets of comparisons in three styles indicated by different colors. In each set, pose description is listed on the left and the images generated by our model, ControlNet, and LDM are listed side by side on the right. "head", "gaze" and directional words are highlighted for better visualization. While images from ControlNet and LDM are meaningful, they often fail to match the head pose, gaze direction, or both specified in the text. Our approach effectively captures and aligns head pose and gaze details with the textual descriptions, maintaining high image quality.
• Figure 4: Visualization of generated images from Label-Guided Model and Sketch-Guided Model. The Label-Guided Model (LGM) falls short of accurately reconstructing true geometric information. On the other hand, the Sketch-Guided Model (SGM) effectively maintains geometric integrity throughout the generation process.