Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning

TL;DR

Pangu-Agent introduces a generalist agent framework that embeds structured, memory-driven reasoning (intrinsic functions) alongside traditional policy execution (extrinsic functions), addressing generalization and data efficiency gaps in RL. It formalizes a nested optimization where intrinsic memory transforms augment the RL objective, and demonstrates how LLM priors can be fine-tuned via supervised learning and PPO-based RL to achieve strong cross-domain performance. Empirical results across ALFWorld, BabyAI, GSM8K, and multi-agent tasks show that composite reasoning and targeted fine-tuning significantly boost success rates and enable cross-domain transfer from a single LLM. The work also provides practical engineering approaches for faster generation, improved credit assignment, and modular prompting, laying groundwork for end-to-end differentiable, tool-enhanced generalist agents.

Abstract

A key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL). However, constructing a standalone RL policy that maps perception to action directly encounters severe problems, chief among them being its lack of generality across multiple tasks and the need for a large amount of training data. The leading cause is that it cannot effectively integrate prior information into the perception-action cycle when devising the policy. Large language models (LLMs) emerged as a fundamental way to incorporate cross-domain knowledge into AI agents but lack crucial learning and adaptation toward specific decision problems. This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies. Our methodology is motivated by the modularity found in the human brain. The framework utilises the construction of intrinsic and extrinsic functions to add previous understandings of reasoning structures. It also provides the adaptive ability to learn models inside every module or function, consistent with the modular structure of cognitive processes. We describe the framework in-depth and compare it with other AI pipelines and existing frameworks. The paper explores practical applications, covering experiments that show the effectiveness of our method. Our results indicate that AI agents perform and adapt far better when organised reasoning and prior knowledge are embedded. This opens the door to more resilient and general AI agent systems.

Figures (4)

• Figure 1: Pictorial depiction of the Pangu-Agent pipeline with RL. Starting from the system prompt and initial state, our agent executes actions in the environment and observes the next state and the reward. The trajectories generated can be used for finetuning the LLM.
• Figure 2: Visualisation of three intrinsic functions demonstrating our formulation's importance in improving our agent's modularity and flexibility. The intrinsic functions can be re-defined and re-configured by users, e.g., $\mu_{1}(\cdot)$ taking an LLM as input to produce thoughts or $\mu_2 (\cdot)$ utilising tools to help improve reasoning. We also support nesting those intrinsic functions to build more general modules for complex and challenging decision-making tasks.
• Figure 3: Visualisation of one example of structured reasoning using nesting of intrinsic and extrinsic functions. The agent initially updates its internal memory, using $\mu_0$, by perceiving its observation. Then the intrinsic function $\mu_1$ selects between Think-and-Act or just Act. The last intrinsic function $\mu_2$ either generates a thought if $\mu_1$ previously selected Think-and-Act otherwise it is null. Finally, the extrinsic function $\pi$ selects the action that the agent will perform in the environment.
• Figure 4: The figure above presents a pictorial depiction of the main components of our proposed agent. On the far left, we can set up a multi-agent environment where each agent can interact with the environment and communicate with other agents. Each agent can be fine-tuned via reinforcement or supervised learning. Each of those agents can support any nesting of intrinsic functions $\Vec{\bm{\mu}}(\cdot)$, such as tool usage, thinking processes, reflecting, planning and others. Those operate on the memory component before producing an action to the outer world via extrinsic processes. Our agent also allows for different prompting strategies and open-source language models, further enabling rigorous experimentation protocols, see \ref{['sec:evaluation']}.