Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions

TL;DR

Recipe2Plan introduces a time-constrained multitasking benchmark for LLM agents using realistic cooking tasks to study the balance between efficiency and feasibility. The framework defines action durations, concurrency, resource constraints, and time-between-action constraints, and evaluates open- and closed-source models with ReAct-style prompting and oracle variants. Key findings show that while LLMs can generate feasible plans without time constraints (e.g., GPT-4o achieving high feasibility), enforcing time constraints sharply reduces success (as low as 21.5%), revealing global planning as the main bottleneck. The work highlights the need for improved temporal reasoning and global planning in LLM agents and provides open-source data and baselines for future development.

Abstract

While Large Language Model-based agents have demonstrated substantial progress in task completion, existing evaluation benchmarks tend to overemphasize single-task performance, with insufficient attention given to the crucial aspects of multitask planning and execution efficiency required in real-world scenarios. To bridge this gap, we present Recipe2Plan, a novel benchmark framework based on real-world cooking scenarios. Unlike conventional benchmarks, Recipe2Plan challenges agents to optimize cooking time through parallel task execution while respecting temporal constraints i.e. specific actions need to be performed within a particular time intervals following the preceding steps. Overly aggressive local parallelization may disrupt this constraint, potentially compromising the entire cooking process. This strict time constraint between actions raises a unique challenge for agents to balance between maximizing concurrent operations and adhering to critical timing constraints. Extensive experiments with state-of-the-art models reveal challenges in maintaining this balance between efficiency and feasibility. The results highlight the need for improved temporal awareness and global multitasking capabilities in large language models. We open-source our benchmark and code at https://github.com/WilliamZR/Recipe2Plan.

Figures (6)

• Figure 1: A simplified demonstration of our benchmark. Actions will either occupy the agent or leave it idle. The four steps of brewing must be executed sequentially as time constraints between actions. The goal for the agent is to plan multitasking to complete the recipes in the shortest time possible without violating any constraints. The first plan illustrates a scenario where the agent attempts always to keep the agent occupied for higher efficiency, resulting in violations of time constraints. The second plan maintains the balance between the efficiency and feasibility of the plan by leaving the agent idle on purpose to maintain the time constraints for all actions.
• Figure 2: Demonstration of relative efficiency. The efficiency is affected by the progress rate and we use relative efficiency (R-Efficiency) to calibrate the metric.
• Figure 3: Results of GPT-4o planning with different priority. Balanced Priority: Blend feasibility and efficiency as in §\ref{['sec3:experiments']}. Feasibility Priority: Only focus on feasibility without considering efficiency.
• Figure 4: Results of prompting GPT-4o under Oracle setting: gold constraints, and Oracle + Hint setting: gold constraints and executable actions at each step.
• Figure 5: Analysis of the distribution of invalid actions and failure source of ReAct + Oracle agents planning with time constraints. GPT-4o + Hint: We add all the executable actions in the prompt to help the agent choose the next action during dynamic local planning.