LLM-BABYBENCH: Understanding and Evaluating Grounded Planning and Reasoning in LLMs

TL;DR

LLM-BabyBench tackles the challenge of evaluating grounded planning and reasoning in LLMs by overlaying a textual interface on a BabyAI-like grid world. It defines three core tasks—Predict, Plan, and Decompose—with datasets generated from expert traces and a standardized evaluation harness that includes environment-interaction validation. Baseline experiments across multiple large language models and prompting strategies show meaningful difficulty, especially as task complexity grows, while prompting methods like Tree-of-Thought offer performance gains and an upper bound is provided by an omniscient OmniBot. The framework is designed to be extensible to additional levels and formats, enabling reproducible, fine-grained assessment of grounded reasoning and guiding progress toward more capable language-guided agents in interactive environments.

Abstract

Assessing the capacity of Large Language Models (LLMs) to plan and reason within the constraints of interactive environments is crucial for developing capable AI agents. We introduce $\textbf{LLM-BabyBench}$, a new benchmark suite designed specifically for this purpose. Built upon a textual adaptation of the procedurally generated BabyAI grid world, this suite evaluates LLMs on three fundamental aspects of grounded intelligence: (1) predicting the consequences of actions on the environment state ($\textbf{Predict}$ task), (2) generating sequences of low-level actions to achieve specified objectives ($\textbf{Plan}$ task), and (3) decomposing high-level instructions into coherent subgoal sequences ($\textbf{Decompose}$ task). We detail the methodology for generating the three corresponding datasets ($\texttt{LLM-BabyBench-Predict}$, $\texttt{-Plan}$, $\texttt{-Decompose}$) by extracting structured information from an expert agent operating within the text-based environment. Furthermore, we provide a standardized evaluation harness and metrics, including environment interaction for validating generated plans, to facilitate reproducible assessment of diverse LLMs. Initial baseline results highlight the challenges posed by these grounded reasoning tasks. The benchmark suite, datasets, data generation code, and evaluation code are made publicly available ($\href{https://github.com/choukrani/llm-babybench}{\text{GitHub}}$, $\href{https://huggingface.co/datasets/salem-mbzuai/LLM-BabyBench}{\text{HuggingFace}}$).

Figures (12)

• Figure 1: Success Rate of the Models on Varying Grid Sizes
• Figure 2: CR, PR and ACI for all models, for different classes of missions.
• Figure 3: An example of custom environment specifically designed for Plan task, with an 32x32 grid size and 180 objects: CustomBabyAI-GoToRedBall-Ultra-180Dists-v0
• Figure 4: BabyAI Unlock level with seed 63. The mission is to "open the green door". This example illustrates why some levels of the original BabyAI platform are unsolvable, and why they were removed from LLM-BabyBench.
• Figure 5: BabyAI UnblockPickup level with seed 8. The mission is to "pickup the green ball". This example provides two type of cases where the OmniBot needs to dynamically add one or many subgoals ot the stack: opening a door and moving an object that prevents from using a door.