Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic AI Needs a Systems Theory

Erik Miehling, Karthikeyan Natesan Ramamurthy, Kush R. Varshney, Matthew Riemer, Djallel Bouneffouf, John T. Richards, Amit Dhurandhar, Elizabeth M. Daly, Michael Hind, Prasanna Sattigeri, Dennis Wei, Ambrish Rawat, Jasmina Gajcin, Werner Geyer

TL;DR

The paper argues that agentic AI risks and capabilities cannot be understood by examining models in isolation; a systemic perspective is needed due to emergent behaviors at interfaces. It develops a systems-theoretic framework centered on functional agency, drawing on psychology, neuroscience, sociology, and biology to explain how simple components can yield advanced system-level capabilities. Contributions include formalizing functional agency, articulating three emergence pathways (environmental embodiment, causal reasoning from prediction, metacognitive shared representations), and discussing open challenges and governance guidance. Significance lies in enabling safer, more controllable agentic systems and preparing for future multimodal, embodied agents by focusing on interaction dynamics and system-level constraints.

Abstract

The endowment of AI with reasoning capabilities and some degree of agency is widely viewed as a path toward more capable and generalizable systems. Our position is that the current development of agentic AI requires a more holistic, systems-theoretic perspective in order to fully understand their capabilities and mitigate any emergent risks. The primary motivation for our position is that AI development is currently overly focused on individual model capabilities, often ignoring broader emergent behavior, leading to a significant underestimation in the true capabilities and associated risks of agentic AI. We describe some fundamental mechanisms by which advanced capabilities can emerge from (comparably simpler) agents simply due to their interaction with the environment and other agents. Informed by an extensive amount of existing literature from various fields, we outline mechanisms for enhanced agent cognition, emergent causal reasoning ability, and metacognitive awareness. We conclude by presenting some key open challenges and guidance for the development of agentic AI. We emphasize that a systems-level perspective is essential for better understanding, and purposefully shaping, agentic AI systems.

Agentic AI Needs a Systems Theory

TL;DR

The paper argues that agentic AI risks and capabilities cannot be understood by examining models in isolation; a systemic perspective is needed due to emergent behaviors at interfaces. It develops a systems-theoretic framework centered on functional agency, drawing on psychology, neuroscience, sociology, and biology to explain how simple components can yield advanced system-level capabilities. Contributions include formalizing functional agency, articulating three emergence pathways (environmental embodiment, causal reasoning from prediction, metacognitive shared representations), and discussing open challenges and governance guidance. Significance lies in enabling safer, more controllable agentic systems and preparing for future multimodal, embodied agents by focusing on interaction dynamics and system-level constraints.

Abstract

The endowment of AI with reasoning capabilities and some degree of agency is widely viewed as a path toward more capable and generalizable systems. Our position is that the current development of agentic AI requires a more holistic, systems-theoretic perspective in order to fully understand their capabilities and mitigate any emergent risks. The primary motivation for our position is that AI development is currently overly focused on individual model capabilities, often ignoring broader emergent behavior, leading to a significant underestimation in the true capabilities and associated risks of agentic AI. We describe some fundamental mechanisms by which advanced capabilities can emerge from (comparably simpler) agents simply due to their interaction with the environment and other agents. Informed by an extensive amount of existing literature from various fields, we outline mechanisms for enhanced agent cognition, emergent causal reasoning ability, and metacognitive awareness. We conclude by presenting some key open challenges and guidance for the development of agentic AI. We emphasize that a systems-level perspective is essential for better understanding, and purposefully shaping, agentic AI systems.

Paper Structure

This paper contains 14 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Defining Agentic Systems
  3. Agency
  4. Agentic Systems
  5. Mechanisms of Emergence
  6. Environment enhances cognition
  7. Ability to predict enables reasoning
  8. Prediction and interaction enables metacognition
  9. Open Challenges
  10. Building generalist agents
  11. Designing efficient agent-agent interactions
  12. Controlling emergence of subgoals
  13. Governing human-agent interactions
  14. Closing Remarks

Figures (1)

  • Figure 1: An agentic system. The human user is responsible for seeding the initial task description and providing any feedback (in the form of clarification or approval) during the solution process. Each agent is described by an LLM or an LMM (large multimodal model), with access to tools that facilitate interaction with the external environment via actions (generated via instructions from the LLM/LMM) and observations (generating LLM/LMM-readable signals). These signals inform the agent's outcome model and drive any necessary adaptation. Agents additionally interact with other agents, communicating any relevant information about the task or observations from the environment.

Theorems & Definitions (1)

  • Definition 2.1: Functional agency