LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments

TL;DR

LangSuitE tackles the challenge of evaluating large language models as embodied agents without relying on simulators. By providing a simulation-free, text-based embodied world with six tasks, it enables flexible experimentation across single- and multi-agent settings, including human-in-the-loop interactions. The authors introduce EmMem, a chain-of-thought–based embodied memory prompting scheme that prompts the model to summarize current embodied state before planning, improving performance especially in low-level action regimes. Benchmark results reveal meaningful improvements from EmMem and reflective prompting, while highlighting ongoing gaps to fully supervised fine-tuned systems and the need for further exploration of embodied generalists and multimodal integrations. Overall, LangSuitE offers a scalable, adaptable testbed that fosters progress toward robust embodied reasoning and coordination with language models.

Abstract

Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents' capacity to develop ``internalized world knowledge'' with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuitE represents a significant step toward building embodied generalists in the context of language models.

Figures (4)

• Figure 1: Task illustration of LangSuit$\cdot$E. Top-Left: a typical example of a cooperative task between two agents. The agents are entirely blind (i.e., without visual perception) and can only obtain embodied information from the system. Around: Exemplar tasks supported by LangSuit$\cdot$E.
• Figure 2: Comparison of our Embodied Environment Description with ALFWorld ALFWorld20.
• Figure 3: A case study for four prompt strategies on Household. Act: The model predicts the next action only. ReAct+EmMem: Our EmMem strategy with ReAct strategy (Reason+Act) yao2022react. Reflexion+EmMem: Our EmMem strategy with Reflexion strategy shinn2023reflexion (1 additional trail of verbal reflection). Because Reflexion will try an additional trail if the task fails and the summarization of the whole last trail will be input to the new trail planning through the Task prompt, we do not show the case in this figure. More details and the prompt instructions for these strategies can be found in Appendix \ref{['app:prompt']}.
• Figure 4: Demonstration of human communications.