LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editing

TL;DR

LaTo tackles the challenge of fine-grained, identity-preserving face editing under substantial pose and expression changes by introducing a landmark-tokenized diffusion transformer. It replaces dense, pixel-wise landmark conditioning with a landmark tokenizer and a location-mapping positional encoding, enabling efficient, geometry-aware interaction with appearance through a unified multi-modal token fuser. A landmark predictor based on a vision–language model provides robust, instruction-driven landmark estimation, bridged by a structured chain-of-thought training regime. The approach is supported by HFL-150K, a large-scale benchmark of over $150{,}000$ editing pairs, and achieves state-of-the-art gains in identity preservation ($7.8\%$) and semantic consistency ($4.6\%$), advancing practical controllable face editing for digital humans and avatars.

Abstract

Recent multimodal models for instruction-based face editing enable semantic manipulation but still struggle with precise attribute control and identity preservation. Structural facial representations such as landmarks are effective for intermediate supervision, yet most existing methods treat them as rigid geometric constraints, which can degrade identity when conditional landmarks deviate significantly from the source (e.g., large expression or pose changes, inaccurate landmark estimates). To address these limitations, we propose LaTo, a landmark-tokenized diffusion transformer for fine-grained, identity-preserving face editing. Our key innovations include: (1) a landmark tokenizer that directly quantizes raw landmark coordinates into discrete facial tokens, obviating the need for dense pixel-wise correspondence; (2) a location-mapping positional encoding that integrates facial and image tokens for unified processing, enabling flexible yet decoupled geometry-appearance interactions with high efficiency and strong identity preservation; and (3) a landmark predictor that leverages vision-language models to infer target landmarks from instructions and source images, whose structured chain-of-thought improves estimation accuracy and interactive control. To mitigate data scarcity, we curate HFL-150K, to our knowledge the largest benchmark for this task, containing over 150K real face pairs with fine-grained instructions. Extensive experiments show that LaTo outperforms state-of-the-art methods by 7.8% in identity preservation and 4.6% in semantic consistency. Code and dataset will be made publicly available upon acceptance.

Figures (14)

• Figure 1: Landmark tokenization in LaTo preserves identity and produces natural results, whereas pixelwise alignment baselines rigidly follow the rendered landmark image and often lose identity under substantial differences in pose, expression, or face shape.
• Figure 2: LaTo enables fine-grained facial expression editing, parametric head-pose editing, or their combination. The small images visualize generated landmarks via landmark predictor, enabling intuitive control signal acquisition.
• Figure 3: Data collection pipeline and statistics of HFL-150K. (a) Synthetic data generation via advanced editing models. (b) Real image pair extraction from video datasets. (c) Expression distribution across 7 categories. (d) Head pose angles aligned with 30° motion budgets.
• Figure 4: Overview of LaTo. The landmark predictor infers target landmarks from source image and instruction via structured chain of thought. A landmark tokenizer and visual VAE encode predicted landmarks and source image into tokens. A token fuser merges tokens with stochastic noise and feeds them to DiT blocks, whose denoising yields a target image aligned with landmarks and instruction.
• Figure 5: Qualitative comparison with state-of-the-art image editing methods.