Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LEGENT: Open Platform for Embodied Agents

Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun

TL;DR

LEGENT addresses the bottleneck of open scalable data for language-grounded embodied AI by providing an open 3D environment and a large-scale data generation pipeline. The approach integrates LLMs and LMMs with embodied supervision to produce diverse scenes, tasks, and trajectories, enabling end-to-end training of vision-language-action models. Experimental results indicate a LEGENT-trained model can outperform GPT-4V on embodied tasks and generalize to unseen settings, illustrating substantial potential for generalizable embodied AI. The platform aims to catalyze community progress by providing accessible tools and documentation for scaling both data and models.

Abstract

Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities.

LEGENT: Open Platform for Embodied Agents

TL;DR

LEGENT addresses the bottleneck of open scalable data for language-grounded embodied AI by providing an open 3D environment and a large-scale data generation pipeline. The approach integrates LLMs and LMMs with embodied supervision to produce diverse scenes, tasks, and trajectories, enabling end-to-end training of vision-language-action models. Experimental results indicate a LEGENT-trained model can outperform GPT-4V on embodied tasks and generalize to unseen settings, illustrating substantial potential for generalizable embodied AI. The platform aims to catalyze community progress by providing accessible tools and documentation for scaling both data and models.

Abstract

Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities.
Paper Structure (13 sections, 7 figures, 3 tables)

This paper contains 13 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LEGENT
  4. Scene
  5. Agent
  6. Interface
  7. Data Generation
  8. Scene Generation
  9. Task Generation
  10. Trajectory Generation
  11. Prototype Experiments
  12. Demo of LEGENT
  13. Conclusion and Future Work

Figures (7)

  • Figure 1: Interaction with the embodied agent in LEGENT. These sequential interactions showcase the agent's ability to answer the user's questions and follow the user's instructions.
  • Figure 2: Features of LEGENT.
  • Figure 3: Examples of importing external assets: user-supplied assets (left); existing datasets (middle); assets generated by generative models (right).
  • Figure 4: An example of humanoid animations, demonstrating accurate object grasping and body movement through spatial planning and inverse kinematics.
  • Figure 5: Examples of generated scenes.
  • ...and 2 more figures