Software Engineering Agents for Embodied Controller Generation : A Study in Minigrid Environments

TL;DR

This work evaluates Software Engineering Agents (SWE-Agents) on embodied controller generation within Minigrid, extending Mini-SWE-Agent to 20 tasks and a 2×2 information-access design that varies environment code access and interactive exploration. It introduces a two-level agency where a code-agent generates Python controllers that are executed in the embodied environment, and it uses 30 independent trials per task to quantify performance via the best@5 metric. Key findings show that fully observable environments allow solving almost all tasks with full access, while partial observability substantially challenges SWE-Agents, underscoring the importance of interactive exploration over static code analysis. The study also documents early hacking behaviors and demonstrates safety measures to prevent illegitimate solutions, establishing a baseline methodology and highlighting embodied tasks as a crucial benchmark for future SWE-Agent research with practical implications for grounded, autonomous software agents. $r_T = 1 - \frac{T}{T_{max}}$ and $\text{best@}5$ are among the mathematical concepts used to quantify rewards and performance, illustrating the rigorous evaluation framework for embodied reasoning systems.

Abstract

Software Engineering Agents (SWE-Agents) have proven effective for traditional software engineering tasks with accessible codebases, but their performance for embodied tasks requiring well-designed information discovery remains unexplored. We present the first extended evaluation of SWE-Agents on controller generation for embodied tasks, adapting Mini-SWE-Agent (MSWEA) to solve 20 diverse embodied tasks from the Minigrid environment. Our experiments compare agent performance across different information access conditions: with and without environment source code access, and with varying capabilities for interactive exploration. We quantify how different information access levels affect SWE-Agent performance for embodied tasks and analyze the relative importance of static code analysis versus dynamic exploration for task solving. This work establishes controller generation for embodied tasks as a crucial evaluation domain for SWE-Agents and provides baseline results for future research in efficient reasoning systems.

Figures (10)

• Figure 1: Two-level agency structure: a code-agent interacts with a code-environment to generates controller-agents (Python programs) to solve the embodied task.
• Figure 2: Example MiniGrid tasks used in our experiments. Each frame shows a different environment category: navigation (Empty, FourRooms), manipulation (DoorKey, KeyCorridor, BlockedUnlockPickup, LockedRoom), hazard (DynamicObstacles, LavaCrossing), and memory (Memory).
• Figure 3: Best@5 success rate of MSWEA across different tasks and information access conditions in Fully Observable Minigrid.
• Figure 4: Best@5 success rate of MSWEA across different tasks and information access conditions in Partially Observable Minigrid.
• Figure 5: Mean-by-category best@5 success rate in Fully Observable Minigrid