Human-Centric Foundation Models: Perception, Generation and Agentic Modeling

TL;DR

The paper tackles the challenge of modeling digital humans by proposing a taxonomy of Human-centric Foundation Models (HcFMs) that spans perception, generation, unified perception-generation, and agentic capabilities. It surveys state-of-the-art methods across unsupervised and supervised perception, GAN- and diffusion-based generation, unified multimodal yes-no modeling with fixed and extended vocabularies, and two agentic frameworks for embodied tasks. Key contributions include a structured synthesis of techniques, identification of cross-cutting themes, and a roadmap addressing data, representations, interactivity, and ethics. The work aims to guide researchers and practitioners toward robust, versatile, and high-fidelity digital humans and humanoid embodiments in real-world applications.

Abstract

Human understanding and generation are critical for modeling digital humans and humanoid embodiments. Recently, Human-centric Foundation Models (HcFMs) inspired by the success of generalist models, such as large language and vision models, have emerged to unify diverse human-centric tasks into a single framework, surpassing traditional task-specific approaches. In this survey, we present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups: (1) Human-centric Perception Foundation Models that capture fine-grained features for multi-modal 2D and 3D understanding. (2) Human-centric AIGC Foundation Models that generate high-fidelity, diverse human-related content. (3) Unified Perception and Generation Models that integrate these capabilities to enhance both human understanding and synthesis. (4) Human-centric Agentic Foundation Models that extend beyond perception and generation to learn human-like intelligence and interactive behaviors for humanoid embodied tasks. We review state-of-the-art techniques, discuss emerging challenges and future research directions. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.

Figures (5)

• Figure 1: A taxonomy of human-centric foundation models with representative examples.
• Figure 2: Different Frameworks of human-centric perception foundation models. Parameters in modules with are used in downstream tasks. Unsupervised foundation models: (a) Contrastive learning methods; (b) Mask image modeling methods. Supervised foundation models: (c) Multitask supervised pretraining methods; (d) Unified modeling methods.
• Figure 3: Different frameworks of human-centric AIGC foundation models. Unsupervised learning methods: (a) A sampled noise is transformed by a mapping network (M) into a style code that modulates the generator (G) at multiple scales, while a discriminator (D) is trained simultaneously through adversarial learning to refine realism. (b) 3D representations are incorporated along with a neural renderer (R) to enhance spatial consistency. Supervised learning methods: (c) The input image is encoded (E) into a latent code, which the diffusion model (DM) iteratively denoises to reconstruct high-quality outputs with fine-grained control. (d) The input video is compressed and decomposed into space-time latents, where transformers capture spatial-temporal dependencies and enable efficient scaling for video generation.
• Figure 4: Human-centric foundation models for unified perception and generation tasks. (1) Vocabulary-Fixed models with fixed vocabulary introduces modality-specific projection layers or directly employing off-the-shelf human-centric tools. (2) Vocabulary-extended models align enriched human-centric representations, e.g., pose parameters, SMPL representations, motion sequences, emotions, and audio signals with the original vocabulary of foundational LLMs.
• Figure 5: Frameworks of human-centric agentic foundation models to endow humanoid robots. (a) Vision-Language-based models: Use pretrained vision-language to learn multi-skill policy; (b) Vision-Language-Action-based models: Finetune a vision-language-action model to generate actions.