The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

Abstract

Latent space is rapidly emerging as a native substrate for language-based models. While modern systems are still commonly understood through explicit token-level generation, an increasing body of work shows that many critical internal processes are more naturally carried out in continuous latent space than in human-readable verbal traces. This shift is driven by the structural limitations of explicit-space computation, including linguistic redundancy, discretization bottlenecks, sequential inefficiency, and semantic loss. This survey aims to provide a unified and up-to-date landscape of latent space in language-based models. We organize the survey into five sequential perspectives: Foundation, Evolution, Mechanism, Ability, and Outlook. We begin by delineating the scope of latent space, distinguishing it from explicit or verbal space and from the latent spaces commonly studied in generative visual models. We then trace the field's evolution from early exploratory efforts to the current large-scale expansion. To organize the technical landscape, we examine existing work through the complementary lenses of mechanism and ability. From the perspective of Mechanism, we identify four major lines of development: Architecture, Representation, Computation, and Optimization. From the perspective of Ability, we show how latent space supports a broad capability spectrum spanning Reasoning, Planning, Modeling, Perception, Memory, Collaboration, and Embodiment. Beyond consolidation, we discuss the key open challenges, and outline promising directions for future research. We hope this survey serves not only as a reference for existing work, but also as a foundation for understanding latent space as a general computational and systems paradigm for next-generation intelligence.

Figures (8)

• Figure 1: Overview of the latent space methods classified by two axes: four main Mechanisms (Section \ref{['sec:mechanism']}) and seven key Abilities (Section \ref{['sec:ability']}). Within our classification system, a single method may be affiliated with one or more mechanisms and capabilities. For the visualization in this figure, we adopt the most appropriate classification for each method; a comprehensive elaboration of these categories will be presented in the main text.
• Figure 2: Outline of the survey, including five sections and sequential questions: Foundation: What is Latent Space? (Section \ref{['sec:foundation_sec2']}), Evolution: How Did Latent Space Develop? (Section \ref{['sec:evolution']}), Mechanism: How Does Latent Space Work? (Section \ref{['sec:mechanism']}), Ability: What Does Latent Space Enable? (Section \ref{['sec:ability']}), and Outlook: What is Next? (Section \ref{['sec:outlook']})
• Figure 3: Comparison of the explicit space and latent space of the language models, including their representational properties and functional capabilities.
• Figure 4: Timeline of representative works in the evolution of latent space research, organized into four developmental stages: Prototype (Section \ref{['sec:embryonic']}), Formation (Section \ref{['sec:foundation']}), Expansion (Section \ref{['sec:expansion']}), and Outbreak (Section \ref{['sec:outbreak']}) stages, where the horizontal axis denotes the month, and vertical axis indicates the number of the latent-level works.
• Figure 5: Representative works operate in accordance with latent space mechanisms. We classify all methods into four lines based on diverse ways of utilizing the latent space, including: Architecture (Section \ref{['sec:architecture']}), Representation (Section \ref{['sec:representation']}), Computation (Section \ref{['sec:computation']}), and Optimization (Section \ref{['sec:optimization']}).