Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulation

TL;DR

The paper tackles the challenge of governing agentic AI in complex sociotechnical systems by separating the information flow from the interaction network and embedding adaptive governance at design time. It introduces a two-layer framework where fixed agent interactions occur on a static graph while a DRL-based RL manager dynamically modulates what information each agent can access, treating the environment as a POMDP and optimizing social welfare with an Actor-Critic approach. Micro-level validation with LLM-based agents demonstrates nuanced strategic behavior and the effectiveness of information-based governance, while experiments in a repeated Prisoner’s Dilemma show the RL manager substantially improves cooperation and overall welfare compared to static baselines. The work highlights the potential for scalable, non-invasive governance of autonomous agents by manipulating information provenance and transparency, with implications for engineering resilient, cooperative multi-agent AI systems in real-world settings.

Abstract

Modern engineered systems increasingly involve complex sociotechnical environments where multiple agents, including humans and the emerging paradigm of agentic AI powered by large language models, must navigate social dilemmas that pit individual interests against collective welfare. As engineered systems evolve toward multi-agent architectures with autonomous LLM-based agents, traditional governance approaches using static rules or fixed network structures fail to address the dynamic uncertainties inherent in real-world operations. This paper presents a novel framework that integrates adaptive governance mechanisms directly into the design of sociotechnical systems through a unique separation of agent interaction networks from information flow networks. We introduce a system comprising strategic LLM-based system agents that engage in repeated interactions and a reinforcement learning-based governing agent that dynamically modulates information transparency. Unlike conventional approaches that require direct structural interventions or payoff modifications, our framework preserves agent autonomy while promoting cooperation through adaptive information governance. The governing agent learns to strategically adjust information disclosure at each timestep, determining what contextual or historical information each system agent can access. Experimental results demonstrate that this RL-based governance significantly enhances cooperation compared to static information-sharing baselines.

Figures (8)

• Figure 1: Overview of the general framework. The framework includes two main entities: the LLM agents and the RL manager. 1) LLM agents receive prompts that describe pairwise strategic games (payoff matrix, objectives, and additional information from the RL manager) and then make strategic decisions like cooperation or defection. Multiple LLM agents are placed in a random network, with connections initialized in each run (and fixed throughout the steps). LLM agents may make different decisions in different interactions based on varying information received. Prompts are refined through micro-level validation for consistent behavior. 2) The RL manager acts as a system manager, observing LLM agents and dynamically determining their information levels to maximize social welfare.
• Figure 2: The diagram illustrates how LLM agents interact with each other. The example prompt shown here includes only the information about the last action. For other experimental settings that involve different types of information, additional details will be incorporated into the highlighted (red) sections of the prompts. The LLM agents interact over several rounds, and after collecting their responses, we can compute the cooperation rate under various scenarios with different information conditions. This cooperation rate serves as a key metric for quantifying both the agents’ cooperative behaviors and the overall system performance.
• Figure 3: Visual illustration of the Win–Stay, Lose–Shift (WSLS) strategy. Player 1 (light blue) adjusts behavior based on the outcome of the previous round against Player 2 (yellow). When both players cooperate, the outcome is registered as a win, and Player 1 continues to cooperate. When Player 1 is exploited, the strategy registers a loss, prompting a shift to defection. Conversely, after successfully exploiting Player 2, the strategy records a win, and Player 1 maintains defection. Finally, when both players defect, the outcome is registered as a loss, and Player 1 switches to cooperation.
• Figure 4: Comparison of cooperation rates over time between RL and baseline methods. The baseline methods utilize specific information during the game, including: "LA" (last action pairs of both LLM agents), "LA+NR" (last action pairs of both LLM agents and the overall cooperation ratio of an LLM agent and also of its neighbor agents), and "LA+AR" (last action pairs and overall cooperation ratio of both LLM agents). The results represent averages from 50 runs, with the shaded areas indicating standard deviation. The results of social welfare in the system over time follow similar trends.
• Figure 5: This analysis tracks the evolution of the LLM agent's behavior over time using different methodologies. The Y-axis displays the percentage of different action pairs resulting from interactions. We categorize 'CD' and 'DC' pairs together because they are symmetric and represent equivalent behaviors. The displayed results are averages derived from 50 runs.