Building LLM Agents by Incorporating Insights from Computer Systems

TL;DR

The paper tackles the lack of principled design in LLM-driven agents by advocating cross-pollination with computer systems. It introduces a von Neumann-inspired modular framework $F=(P,C,M,T,A)$ where perception, cognition, memory, tools, and action interact in a closed loop, with a time-step rule $a_t = A\bigl( C\bigl( P(o_1, a_1, \dots, o_t), M_r, T_c \bigr) \bigr)$. Contributions include a systematic comparison between LLM agents and computer architectures, identification of memory hierarchies, memory read/write mechanisms, tool management, and computation parallels, plus a roadmap of future directions such as memory refinement, multi-core cognition, parallelization, and learning mechanisms beyond computers. The framework aims to provide a principled basis for scalable, interpretable, and adaptable LLM agents and guides experimental validation for open-world deployment.

Abstract

LLM-driven autonomous agents have emerged as a promising direction in recent years. However, many of these LLM agents are designed empirically or based on intuition, often lacking systematic design principles, which results in diverse agent structures with limited generality and scalability. In this paper, we advocate for building LLM agents by incorporating insights from computer systems. Inspired by the von Neumann architecture, we propose a structured framework for LLM agentic systems, emphasizing modular design and universal principles. Specifically, this paper first provides a comprehensive review of LLM agents from the computer system perspective, then identifies key challenges and future directions inspired by computer system design, and finally explores the learning mechanisms for LLM agents beyond the computer system. The insights gained from this comparative analysis offer a foundation for systematic LLM agent design and advancement.

Figures (4)

• Figure 1: An analogy between von Neumann architecture and LLM agent architecture with task execution workflows (\ref{['Appdenix:A']}).
• Figure 2: A framework diagram of LLM agents inspired by the von Neumann architecture. This framework illustrates the different components of LLM agents and specific classifications within each module, along with the interactions between the modules.
• Figure 3: An intuitive comparison between computer memory hierarchy and agent memory hierarchy.
• Figure 4: A design decision diagram illustrating the principles of building LLM agents.