REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks
Longling Geng, Edward Y. Chang
TL;DR
<3-5 sentence high-level summary>REALM-Bench addresses the lack of standardized, realistic benchmarks for evaluating AI planning in real-world multi-agent settings with disruptions. It combines two problem families (P-series planning and JSSP) into 14 scalable scenarios, enabling assessment of planning quality, coordination, and disruption adaptation across LLMs and agent frameworks. The framework provides standardized metrics, extensible implementation, and public leaderboards, highlighting current capabilities and gaps, particularly in dynamic coordination and long-lived workflows. This benchmark has practical impact by pushing toward more robust, adaptable planning systems for logistics, scheduling, and crisis-response applications.
Abstract
This benchmark suite provides a comprehensive evaluation framework for assessing both individual LLMs and multi-agent systems in Real-world planning and scheduling scenarios. The suite encompasses 14 designed planning and scheduling problems that progress from basic to highly complex, incorporating key aspects such as multi-agent coordination, inter-agent dependencies, and dynamic environmental disruptions. Each problem can be scaled along three dimensions: the number of parallel planning threads, the complexity of inter-dependencies, and the frequency of unexpected disruptions requiring Real-time adaptation. The benchmark includes 14 detailed problem specifications, 15 comparison methods including Random, LPT, SPT, STPT, MPSR, DRL-Liu, GP, GEP, LSO, SPT/TWKR, DRL-Chen, DRL-Zhang, 2+ evaluation metrics, and baseline implementations using 3+ LLMs including GPT-4o, Claude-3.7, DeepSeek-R1, and 4 contemporary frameworks including LangGraph, AutoGen, CrewAI, and Swarm, enabling rigorous testing of both single-agent and multi-agent planning capabilities. Through standardized evaluation criteria and scalable complexity, this benchmark aims to be opened to public, and drive progress in developing more adaptable, robust, and scalable AI planning systems for Real-world applications.