Focal-RegionFace: Generating Fine-Grained Multi-attribute Descriptions for Arbitrarily Selected Face Focal Regions

TL;DR

This work defines FaceFocalDesc, a task to generate and recognize multi-attribute natural language descriptions for arbitrarily selected facial regions, including action units, emotions, and age. It introduces the Multimodal Face Region-Focal Dataset (MFRF) and a four-stage progressive fine-tuning framework, Focal-RegionFace, built on Qwen2.5-VL to achieve region-aware, interpretable facial analysis. Through extensive experiments with both open- and closed-source MLLMs, the authors demonstrate superior region-focused generation quality and multi-attribute recognition accuracy, along with robust ablations validating the stage-wise design. The approach advances fine-grained, region-centered facial understanding with potential applications in cosmetic, medical, and surveillance domains, while acknowledging limitations in annotation bias and evaluator dependence.

Abstract

In this paper, we introduce an underexplored problem in facial analysis: generating and recognizing multi-attribute natural language descriptions, containing facial action units (AUs), emotional states, and age estimation, for arbitrarily selected face regions (termed FaceFocalDesc). We argue that the system's ability to focus on individual facial areas leads to better understanding and control. To achieve this capability, we construct a new multi-attribute description dataset for arbitrarily selected face regions, providing rich region-level annotations and natural language descriptions. Further, we propose a fine-tuned vision-language model based on Qwen2.5-VL, called Focal-RegionFace for facial state analysis, which incrementally refines its focus on localized facial features through multiple progressively fine-tuning stages, resulting in interpretable age estimation, FAU and emotion detection. Experimental results show that Focal-RegionFace achieves the best performance on the new benchmark in terms of traditional and widely used metrics, as well as new proposed metrics. This fully verifies its effectiveness and versatility in fine-grained multi-attribute face region-focal analysis scenarios.

Figures (11)

• Figure 1: Comparison of facial state analysis capabilities among mainstream MLLMs and our model achieve superior performance in all NLP metrics. In particular, we show the detailed results of the traditional facial state recognition method, MLLM Qwen2.5-VL and our Focal-RegionFace model. Our Focal-RegionFace model can generate more detailed multi-attribute facial descriptions of arbitrarily selected face regions.
• Figure 2: Overview of Focal-RegionFace with multi-stage fine-tuning. We first perform global face multi-attribute information-aware fine-tuning of Qwen2.5-VL in Stage-I, including age, emotion and AU recognition. Then, we make the model focus on region-focal reasoning in Stage-II and Stage-III in a progressive fine-tuning manner, thus obtaining a Focal-RegionFace MLLM with fine-grained multi-attribute language interpretation. Next, further multimodal inference fine-tuning (Stage-IV) is carried out based on the multi-region visual understanding results, so that the model develops a fine-grained multimodal multi-attribute recognition capability.
• Figure 3: Ablation: MLLM evaluation
• Figure 4: Visual comparisons of different face state description generators for multiple face attributes, including facial AU, emotion, and age. The red boxes are randomly selected areas. And the descriptions in red are incorrect or region-irrelevant generation. (Blue: AUs description; Green: Muscle description; Purple: Comprehensive analysis of skin details)
• Figure 5: Prompt details for generating fine-grained descriptions of AGE, Emotion and AUs using GPT-4o.