Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Real Barrier to LLM Agent Usability is Agentic ROI

Weiwen Liu, Jiarui Qin, Xu Huang, Xingshan Zeng, Yunjia Xi, Jianghao Lin, Chuhan Wu, Yasheng Wang, Lifeng Shang, Ruiming Tang, Defu Lian, Yong Yu, Weinan Zhang

TL;DR

This paper argues that the main barrier to practical usability of LLM agents in mass-market settings is the ROI of agent use, not just model capability. It introduces Agentic ROI, defined as $Agentic ROI = \\frac{(\\text{Information Quality}-\\tau)\\cdot(\\text{Human Time}-\\text{Agent Time})}{\\text{Interaction Time}\\cdot \\text{Expense}}$, and identifies information quality, agent time, and cost as the three driving factors. It proposes a zigzag development trajectory—initial scaling up to boost information quality, followed by scaling down to reduce time and cost—and outlines a roadmap across pre-training, post-training, test-time, world-model, and robustness domains, plus efficiency strategies. The contribution is a practical, ROI-centric framework to guide the design, evaluation, and deployment of scalable, trustworthy LLM agents for real-world use.

Abstract

Large Language Model (LLM) agents represent a promising shift in human-AI interaction, moving beyond passive prompt-response systems to autonomous agents capable of reasoning, planning, and goal-directed action. Despite the widespread application in specialized, high-effort tasks like coding and scientific research, we highlight a critical usability gap in high-demand, mass-market applications. This position paper argues that the limited real-world adoption of LLM agents stems not only from gaps in model capabilities, but also from a fundamental tradeoff between the value an agent can provide and the costs incurred during real-world use. Hence, we call for a shift from solely optimizing model performance to a broader, utility-driven perspective: evaluating agents through the lens of the overall agentic return on investment (Agent ROI). By identifying key factors that determine Agentic ROI--information quality, agent time, and cost--we posit a zigzag development trajectory in optimizing agentic ROI: first scaling up to improve the information quality, then scaling down to minimize the time and cost. We outline the roadmap across different development stages to bridge the current usability gaps, aiming to make LLM agents truly scalable, accessible, and effective in real-world contexts.

The Real Barrier to LLM Agent Usability is Agentic ROI

TL;DR

This paper argues that the main barrier to practical usability of LLM agents in mass-market settings is the ROI of agent use, not just model capability. It introduces Agentic ROI, defined as , and identifies information quality, agent time, and cost as the three driving factors. It proposes a zigzag development trajectory—initial scaling up to boost information quality, followed by scaling down to reduce time and cost—and outlines a roadmap across pre-training, post-training, test-time, world-model, and robustness domains, plus efficiency strategies. The contribution is a practical, ROI-centric framework to guide the design, evaluation, and deployment of scalable, trustworthy LLM agents for real-world use.

Abstract

Large Language Model (LLM) agents represent a promising shift in human-AI interaction, moving beyond passive prompt-response systems to autonomous agents capable of reasoning, planning, and goal-directed action. Despite the widespread application in specialized, high-effort tasks like coding and scientific research, we highlight a critical usability gap in high-demand, mass-market applications. This position paper argues that the limited real-world adoption of LLM agents stems not only from gaps in model capabilities, but also from a fundamental tradeoff between the value an agent can provide and the costs incurred during real-world use. Hence, we call for a shift from solely optimizing model performance to a broader, utility-driven perspective: evaluating agents through the lens of the overall agentic return on investment (Agent ROI). By identifying key factors that determine Agentic ROI--information quality, agent time, and cost--we posit a zigzag development trajectory in optimizing agentic ROI: first scaling up to improve the information quality, then scaling down to minimize the time and cost. We outline the roadmap across different development stages to bridge the current usability gaps, aiming to make LLM agents truly scalable, accessible, and effective in real-world contexts.

Paper Structure

This paper contains 14 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Zigzag Development Trend in Optimizing Agentic ROI
  3. Scaling Up for Information Quality
  4. Pre-training Scaling
  5. Post-training Scaling
  6. Test-time Scaling
  7. Building World Model
  8. Ensuring Robustness and Security
  9. Scaling Down to Reduce Agent Time and Cost
  10. Reducing Agent Time
  11. Minimizing Cost
  12. Literature Review
  13. Alternative Views and Broader Impacts
  14. Conclusion

Figures (2)

  • Figure 1: Illustration of the usability for LLM agents across different application domains. User demand is estimated based on the approximate monthly active users (MAU) of conventional applications in each domain. Agentic ROI is a conceptual representation of relative trends. The listed agent products are illustrative and may not be exhaustive.
  • Figure 2: The zigzag performance trend of different OpenAI series models. Model size is estimated based on inference cost. Smaller models (e.g., o3-mini, o4-mini) achieve performance comparable to larger predecessors (e.g., o1, o3) from earlier generations.