The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models

TL;DR

This work addresses the challenge that embodied AI must fuse multi-source data across multiple processing stages to sustain stable perception-to-action loops. It introduces the Semantic Lifecycle, a unified framework organizing semantic processing into Acquisition, Representation, and Storage, and analyzes foundation models (LLMs, VFMs, MFMs, GFMs, EFMs) as the driving components across these stages. The paper provides a structured taxonomy and survey of methods for scene-centric and agent-centric acquisition, cross-modal representation alignment, and persistent storage with update mechanisms, highlighting open challenges like end-to-end evaluation and long-horizon consistency. By proposing lifecycle-aware memory and holistic evaluation, the work lays groundwork for robust, continual embodied intelligence capable of lifelong learning in open-world environments.

Abstract

Semantic information in embodied AI is inherently multi-source and multi-stage, making it challenging to fully leverage for achieving stable perception-to-action loops in real-world environments. Early studies have combined manual engineering with deep neural networks, achieving notable progress in specific semantic-related embodied tasks. However, as embodied agents encounter increasingly complex environments and open-ended tasks, the demand for more generalizable and robust semantic processing capabilities has become imperative. Recent advances in foundation models (FMs) address this challenge through their cross-domain generalization abilities and rich semantic priors, reshaping the landscape of embodied AI research. In this survey, we propose the Semantic Lifecycle as a unified framework to characterize the evolution of semantic knowledge within embodied AI driven by foundation models. Departing from traditional paradigms that treat semantic processing as isolated modules or disjoint tasks, our framework offers a holistic perspective that captures the continuous flow and maintenance of semantic knowledge. Guided by this embodied semantic lifecycle, we further analyze and compare recent advances across three key stages: acquisition, representation, and storage. Finally, we summarize existing challenges and outline promising directions for future research.

Figures (3)

• Figure 1: The Semantic Lifecycle Framework in embodied AI. This unified framework characterizes the continuous evolution of semantic knowledge across three interdependent stages: (1) Acquisition, extracting scene-centric and agent-centric semantics from the environment; (2) Representation, aligning heterogeneous semantic into explicit, implicit, or hybrid forms; and (3) Storage, consolidating semantic knowledge into persistent substrates via specific update operators to support the embodied agent.
• Figure 2: A semantic lifecycle-based taxonomy of embodied semantics research, where literature is organized into three main stages: Acquisition, Representation and Storage.
• Figure 3: Chronological overview of representative works in embodied semantic acquisition, organized into five distinct categories.