Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Do Tools and Planning Help LLMs Think? A Cost- and Latency-Aware Benchmark

Subha Ghoshal, Ali Al-Bustami

TL;DR

This work introduces a practical LLM Thinking Benchmark to study when inference-time planning and external tools improve performance under cost and latency constraints. It evaluates two real-world tasks—Event-QA over a knowledge-graph and CMV-based persuasion—using a three-stage plan–execute–replan workflow with LangGraph and task-specific tools, comparing GPT-4o and GPT-4o-mini. Across tasks, planning and tool use boosted Event-QA accuracy but dramatically increased latency, while CMV favored fast one-shot prompts, especially with the smaller model, indicating that benefits of 'thinking' are highly task- and tool-dependent. The authors propose cost-aware deployment guidance—start with fast, lightweight baselines and introduce tooling only when necessary—along with directions for broader evaluation and error analysis in future work.

Abstract

Modern large language models (LLMs) increasingly rely on inference-time planning and external tools to improve reasoning. We benchmark this behavior on two real-world settings: event-centric question answering over graph-structured knowledge (Event-QA) and persuasive response generation in Reddit ChangeMyView (CMV). Using LangChain and LangGraph, we compare a one-shot baseline against a plan-execute-replan agent equipped with task-specific tools (DBpedia SPARQL/lookup/schema exploration, Wikipedia-focused retrieval, and topical web search). We evaluate on 60 examples each from Event-QA and CMV (3 splits of 20), and report both mean end-to-end latency and per-example token cost estimates. We evaluate GPT-4o and GPT-4o-mini under identical workflows and report accuracy and end-to-end latency. On Event-QA, the best tool-augmented configuration improves accuracy (e.g., 47.5\% $\rightarrow$ 67.5\% for GPT-4o) while increasing latency by orders of magnitude ($\sim$8s $\rightarrow$ $\sim$317s per example). On CMV, one-shot prompting is strongest (e.g., GPT-4o-mini achieves 75\% at $\sim$6s), and planning+search increases latency substantially without consistent gains. However, complex multi-tool orchestration exposes failure modes where the smaller model degrades. Overall, the findings highlight the need for task-specific, cost-aware choices of both model size and agent/tooling complexity.

When Do Tools and Planning Help LLMs Think? A Cost- and Latency-Aware Benchmark

TL;DR

This work introduces a practical LLM Thinking Benchmark to study when inference-time planning and external tools improve performance under cost and latency constraints. It evaluates two real-world tasks—Event-QA over a knowledge-graph and CMV-based persuasion—using a three-stage plan–execute–replan workflow with LangGraph and task-specific tools, comparing GPT-4o and GPT-4o-mini. Across tasks, planning and tool use boosted Event-QA accuracy but dramatically increased latency, while CMV favored fast one-shot prompts, especially with the smaller model, indicating that benefits of 'thinking' are highly task- and tool-dependent. The authors propose cost-aware deployment guidance—start with fast, lightweight baselines and introduce tooling only when necessary—along with directions for broader evaluation and error analysis in future work.

Abstract

Modern large language models (LLMs) increasingly rely on inference-time planning and external tools to improve reasoning. We benchmark this behavior on two real-world settings: event-centric question answering over graph-structured knowledge (Event-QA) and persuasive response generation in Reddit ChangeMyView (CMV). Using LangChain and LangGraph, we compare a one-shot baseline against a plan-execute-replan agent equipped with task-specific tools (DBpedia SPARQL/lookup/schema exploration, Wikipedia-focused retrieval, and topical web search). We evaluate on 60 examples each from Event-QA and CMV (3 splits of 20), and report both mean end-to-end latency and per-example token cost estimates. We evaluate GPT-4o and GPT-4o-mini under identical workflows and report accuracy and end-to-end latency. On Event-QA, the best tool-augmented configuration improves accuracy (e.g., 47.5\% 67.5\% for GPT-4o) while increasing latency by orders of magnitude (8s 317s per example). On CMV, one-shot prompting is strongest (e.g., GPT-4o-mini achieves 75\% at 6s), and planning+search increases latency substantially without consistent gains. However, complex multi-tool orchestration exposes failure modes where the smaller model degrades. Overall, the findings highlight the need for task-specific, cost-aware choices of both model size and agent/tooling complexity.
Paper Structure (28 sections, 2 equations, 8 figures, 3 tables)

This paper contains 28 sections, 2 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning and inference-time compute
  4. Tool-augmented and program-aided LLMs
  5. Benchmarks for knowledge graphs and persuasion
  6. Cost- and latency-aware evaluation
  7. Methodology
  8. Evaluated approaches per dataset
  9. Event-QA approaches
  10. ChangeMyView approaches
  11. LangGraph approaches
  12. LangGraph tools
  13. LLMs used
  14. Experimental protocol and tuning
  15. EventQA tuning
  16. ...and 13 more sections

Figures (8)

  • Figure 1: LLM Reasoning Evaluation Workflow
  • Figure 2: The two LangGraph approaches evaluated. Left: a one-shot baseline where the LLM answers directly. Right: a plan--execute--replan pipeline where the LLM plans, invokes tools during execution, and then answers or revises the plan.
  • Figure 3: EventQA - GPT-4o
  • Figure 4: EventQA - GPT-4o-mini
  • Figure 5: EventQA - Best Overall
  • ...and 3 more figures