Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multiagent Reinforcement Learning with Neighbor Action Estimation

Zhenglong Luo, Zhiyong Chen, Aoxiang Liu

TL;DR

The paper tackles multiagent reinforcement learning under communication constraints by introducing an Action Estimation Network (AEN) that predicts other agents' actions from local observations. This module is integrated with TD3 to form AEN-TD3, enabling decentralized learning without explicit action sharing. The approach is validated in a realistic dual-arm robotic manipulation task using Robosuite MuJoCo, with sim-to-real transfer facilitated by signal interpolation and tightened safety constraints. Results show that AEN-TD3 achieves performance comparable to centralized TD3 while reducing reliance on information infrastructure, highlighting practical implications for scalable, infrastructure-light cooperative robotics.

Abstract

Multiagent reinforcement learning, as a prominent intelligent paradigm, enables collaborative decision-making within complex systems. However, existing approaches often rely on explicit action exchange between agents to evaluate action value functions, which is frequently impractical in real-world engineering environments due to communication constraints, latency, energy consumption, and reliability requirements. From an artificial intelligence perspective, this paper proposes an enhanced multiagent reinforcement learning framework that employs action estimation neural networks to infer agent behaviors. By integrating a lightweight action estimation module, each agent infers neighboring agents' behaviors using only locally observable information, enabling collaborative policy learning without explicit action sharing. This approach is fully compatible with standard TD3 algorithms and scalable to larger multiagent systems. At the engineering application level, this framework has been implemented and validated in dual-arm robotic manipulation tasks: two robotic arms collaboratively lift objects. Experimental results demonstrate that this approach significantly enhances the robustness and deployment feasibility of real-world robotic systems while reducing dependence on information infrastructure. Overall, this research advances the development of decentralized multiagent artificial intelligence systems while enabling AI to operate effectively in dynamic, information-constrained real-world environments.

Multiagent Reinforcement Learning with Neighbor Action Estimation

TL;DR

The paper tackles multiagent reinforcement learning under communication constraints by introducing an Action Estimation Network (AEN) that predicts other agents' actions from local observations. This module is integrated with TD3 to form AEN-TD3, enabling decentralized learning without explicit action sharing. The approach is validated in a realistic dual-arm robotic manipulation task using Robosuite MuJoCo, with sim-to-real transfer facilitated by signal interpolation and tightened safety constraints. Results show that AEN-TD3 achieves performance comparable to centralized TD3 while reducing reliance on information infrastructure, highlighting practical implications for scalable, infrastructure-light cooperative robotics.

Abstract

Multiagent reinforcement learning, as a prominent intelligent paradigm, enables collaborative decision-making within complex systems. However, existing approaches often rely on explicit action exchange between agents to evaluate action value functions, which is frequently impractical in real-world engineering environments due to communication constraints, latency, energy consumption, and reliability requirements. From an artificial intelligence perspective, this paper proposes an enhanced multiagent reinforcement learning framework that employs action estimation neural networks to infer agent behaviors. By integrating a lightweight action estimation module, each agent infers neighboring agents' behaviors using only locally observable information, enabling collaborative policy learning without explicit action sharing. This approach is fully compatible with standard TD3 algorithms and scalable to larger multiagent systems. At the engineering application level, this framework has been implemented and validated in dual-arm robotic manipulation tasks: two robotic arms collaboratively lift objects. Experimental results demonstrate that this approach significantly enhances the robustness and deployment feasibility of real-world robotic systems while reducing dependence on information infrastructure. Overall, this research advances the development of decentralized multiagent artificial intelligence systems while enabling AI to operate effectively in dynamic, information-constrained real-world environments.
Paper Structure (14 sections, 6 equations, 5 figures, 1 algorithm)

This paper contains 14 sections, 6 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Action Estimation Network
  4. Update of Networks
  5. Experimental Results
  6. Simulation Environment
  7. Simulation Results and Evaluation
  8. Centralized Setting: TD3
  9. Decentralized Setting: AEN-TD3
  10. Policy Deployment and Experiments
  11. Signal Interpolation
  12. Tightened Safety Constraints
  13. Results and Evaluation
  14. Conclusion

Figures (5)

  • Figure 1: Performance of returns during TD3 training.
  • Figure 2: Performance of returns during AEN-TD3 training.
  • Figure 3: Signal Interpolation Mechanism
  • Figure 4: Sequential snapshots of experiments on the physical robotic platform using policies trained with TD3 (left) and AEN-TD3 (right).
  • Figure 5: Component height trajectories using policies trained with TD3 and AEN-TD3.