Position: Foundation Models Need Digital Twin Representations
Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath
TL;DR
This paper argues that token-based representations underpinning current foundation models fragment the continuity of real-world multimodal data and fail to encode explicit domain knowledge, limiting causal reasoning and cross-modal coherence. It proposes digital twin representations as an outcome-driven, physically grounded alternative that preserves semantic relationships and domain constraints, enabling more efficient data synthesis, better sim-to-real transfer, and improved interpretability. The authors formalize a DT-based framework, distinguish DT representations from token representations, and present five core assumptions on how DTs can enhance knowledge encoding, data synthesis, causality, semantic unity, and transparency. If realized at scale, DT representations could significantly improve robustness, generalization, and reliability of multimodal AI systems across robotics, medicine, and complex vision-language tasks. The work also outlines practical research directions for building scalable DT pipelines, establishing theoretical foundations, and developing hybrid architectures that combine the strengths of both DT and token representations.
Abstract
Current foundation models (FMs) rely on token representations that directly fragment continuous real-world multimodal data into discrete tokens. They limit FMs to learning real-world knowledge and relationships purely through statistical correlation rather than leveraging explicit domain knowledge. Consequently, current FMs struggle with maintaining semantic coherence across modalities, capturing fine-grained spatial-temporal dynamics, and performing causal reasoning. These limitations cannot be overcome by simply scaling up model size or expanding datasets. This position paper argues that the machine learning community should consider digital twin (DT) representations, which are outcome-driven digital representations that serve as building blocks for creating virtual replicas of physical processes, as an alternative to the token representation for building FMs. Finally, we discuss how DT representations can address these challenges by providing physically grounded representations that explicitly encode domain knowledge and preserve the continuous nature of real-world processes.