Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TongSIM: A General Platform for Simulating Intelligent Machines

Zhe Sun, Kunlun Wu, Chuanjian Fu, Zeming Song, Langyong Shi, Zihe Xue, Bohan Jing, Ying Yang, Xiaomeng Gao, Aijia Li, Tianyu Guo, Huiying Li, Xueyuan Yang, Rongkai Liu, Xinyi He, Yuxi Wang, Yue Li, Mingyuan Liu, Yujie Lu, Hongzhao Xie, Shiyun Zhao, Bo Dai, Wei Wang, Tao Yuan, Song-Chun Zhu, Yujia Peng, Zhenliang Zhang

TL;DR

TongSIM introduces a versatile, high-fidelity simulation platform built on Unreal Engine 5.6, featuring 115 indoor scenes plus a city-scale outdoor world, a robust Python API, and scalable parallel training to support diverse embodied AI tasks. It formalizes five benchmark families—single-agent navigation, multi-agent cooperation, human-robot social navigation, and two tiers of composite tasks—spanning perception, planning, social reasoning, and manipulation. Empirical results show current RL and MLLM-based agents excel at perception but struggle with long-horizon planning, spatial reasoning, and social norm compliance, underscoring the need for integrated embodied reasoning. By open-sourcing TongSIM and its benchmarks, the authors aim to accelerate development toward general embodied intelligence and bridge the simulation-to-reality gap across disciplines.

Abstract

As artificial intelligence (AI) rapidly advances, especially in multimodal large language models (MLLMs), research focus is shifting from single-modality text processing to the more complex domains of multimodal and embodied AI. Embodied intelligence focuses on training agents within realistic simulated environments, leveraging physical interaction and action feedback rather than conventionally labeled datasets. Yet, most existing simulation platforms remain narrowly designed, each tailored to specific tasks. A versatile, general-purpose training environment that can support everything from low-level embodied navigation to high-level composite activities, such as multi-agent social simulation and human-AI collaboration, remains largely unavailable. To bridge this gap, we introduce TongSIM, a high-fidelity, general-purpose platform for training and evaluating embodied agents. TongSIM offers practical advantages by providing over 100 diverse, multi-room indoor scenarios as well as an open-ended, interaction-rich outdoor town simulation, ensuring broad applicability across research needs. Its comprehensive evaluation framework and benchmarks enable precise assessment of agent capabilities, such as perception, cognition, decision-making, human-robot cooperation, and spatial and social reasoning. With features like customized scenes, task-adaptive fidelity, diverse agent types, and dynamic environmental simulation, TongSIM delivers flexibility and scalability for researchers, serving as a unified platform that accelerates training, evaluation, and advancement toward general embodied intelligence.

TongSIM: A General Platform for Simulating Intelligent Machines

TL;DR

TongSIM introduces a versatile, high-fidelity simulation platform built on Unreal Engine 5.6, featuring 115 indoor scenes plus a city-scale outdoor world, a robust Python API, and scalable parallel training to support diverse embodied AI tasks. It formalizes five benchmark families—single-agent navigation, multi-agent cooperation, human-robot social navigation, and two tiers of composite tasks—spanning perception, planning, social reasoning, and manipulation. Empirical results show current RL and MLLM-based agents excel at perception but struggle with long-horizon planning, spatial reasoning, and social norm compliance, underscoring the need for integrated embodied reasoning. By open-sourcing TongSIM and its benchmarks, the authors aim to accelerate development toward general embodied intelligence and bridge the simulation-to-reality gap across disciplines.

Abstract

As artificial intelligence (AI) rapidly advances, especially in multimodal large language models (MLLMs), research focus is shifting from single-modality text processing to the more complex domains of multimodal and embodied AI. Embodied intelligence focuses on training agents within realistic simulated environments, leveraging physical interaction and action feedback rather than conventionally labeled datasets. Yet, most existing simulation platforms remain narrowly designed, each tailored to specific tasks. A versatile, general-purpose training environment that can support everything from low-level embodied navigation to high-level composite activities, such as multi-agent social simulation and human-AI collaboration, remains largely unavailable. To bridge this gap, we introduce TongSIM, a high-fidelity, general-purpose platform for training and evaluating embodied agents. TongSIM offers practical advantages by providing over 100 diverse, multi-room indoor scenarios as well as an open-ended, interaction-rich outdoor town simulation, ensuring broad applicability across research needs. Its comprehensive evaluation framework and benchmarks enable precise assessment of agent capabilities, such as perception, cognition, decision-making, human-robot cooperation, and spatial and social reasoning. With features like customized scenes, task-adaptive fidelity, diverse agent types, and dynamic environmental simulation, TongSIM delivers flexibility and scalability for researchers, serving as a unified platform that accelerates training, evaluation, and advancement toward general embodied intelligence.
Paper Structure (42 sections, 4 equations, 14 figures, 7 tables)

This paper contains 42 sections, 4 equations, 14 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Simulators for Embodied AI
  4. Benchmarks for Embodied AI
  5. TongSIM Platform
  6. Platform Overview
  7. High Fidelity Scenes
  8. Customize Tasks and Scenes
  9. Agent
  10. Platform Features
  11. Physical Simulation
  12. Interactive Objects
  13. Interactive Objects
  14. Experimental features
  15. Parallel Training
  16. ...and 27 more sections

Figures (14)

  • Figure 1: TongSIM, a simulation platform for general-purpose embodied AI agent training and evaluation. We provide diverse high-fidelity indoor and outdoor scenes that suit a large range of tasks, as well as multiple embodiments, including not only human-like figures but also robotic ones.
  • Figure 2: Overview of the TongSIM system architecture. The platform contains a unreal engine-based simulator and a python controller which can communicate with the simulator. Within the simulator, four features are supported, including multimodal data sensors, high-fidelity simulation, large-scale NPC systems, and parallel training. Based on this platform, various tasks are developed to support embodied AI, such as indoor navigation, multi-agent cooperation, home composite tasks, etc. Also, the platform integrates the evaluation system to evaluate the performance of agents about tasks, abilities, and level of intelligence.
  • Figure 3: Statistics of indoor environments in TongSIM. The dataset spans diverse functional categories (e.g., residential, commercial) and architectural styles (e.g., modern, classical Chinese), designed to support complex, human-centric tasks.
  • Figure 4: Visualization of the TongSIM Outdoor World. The platform simulates a holistic, spatially contiguous urban environment rather than isolated fragments.
  • Figure 5: Statistics of the objects in TongSIM. The dataset spans diverse functional categories (e.g., residential, commercial) and architectural styles (e.g., modern, classical Chinese), designed to support complex, human-centric tasks.
  • ...and 9 more figures