Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence

TL;DR

Butter-Bench provides a real-world benchmark for evaluating practical intelligence in LLM-controlled robots by isolating the orchestrator from the low-level executor and testing in a TurtleBot-based setup. The study reveals a large gap between current LLMs and human performance, with social understanding and multi-step spatial planning posing the greatest challenges, and shows that fine-tuning embodied reasoning does not reliably improve scores. By combining hardware-software architecture, task decomposition, and red-teaming, the work highlights both the potential and the safety considerations for deploying embodied LLM systems in real environments. The results motivate future work to broaden platforms, environments, and evaluation rigor to better anticipate deployment challenges and necessary guardrails.

Abstract

We present Butter-Bench, a benchmark evaluating large language model (LLM) controlled robots for practical intelligence, defined as the ability to navigate the messiness of the physical world. Current state-of-the-art robotic systems use a hierarchical architecture with LLMs in charge of high-level reasoning, and a Vision Language Action (VLA) model for low-level control. Butter-Bench evaluates the LLM part in isolation from the VLA. Although LLMs have repeatedly surpassed humans in evaluations requiring analytical intelligence, we find humans still outperform LLMs on Butter-Bench. The best LLMs score 40% on Butter-Bench, while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. We also evaluate LLMs that are fine-tuned for embodied reasoning and conclude that this training does not improve their score on Butter-Bench.

Figures (9)

• Figure 1: Hierarchical robot system overview.
• Figure 2: deepmind2025gemini
• Figure 3: turtlebot4
• Figure 4: Overview of Butter-Bench
• Figure 7: Mean task completion rate