A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms

TL;DR

The paper compares centralised PPO and distributed MAPPO for controlling a Cosserat-rod soft arm across varying numbers of controlled sections $n$. By applying identical budgets to both architectures under nominal, disturbance, and actuator-failure scenarios, it shows that distributed control provides higher success rates, resilience, and faster convergence in more complex systems $(n>4)$, while centralised control is more time-efficient to train and can outperform in very simple setups $(n\le 2)$. The study highlights clear trade-offs between collaboration via a central critic and local, agent-level learning with communication, offering actionable guidance for sim-to-real deployment of soft-robot controllers. These insights inform design choices for robust, scalable soft-robot manipulation in real-world environments.

Abstract

This paper presents a quantitative comparison between centralised and distributed multi-agent reinforcement learning (MARL) architectures for controlling a soft robotic arm modelled as a Cosserat rod in simulation. Using PyElastica and the OpenAI Gym interface, we train both a global Proximal Policy Optimisation (PPO) controller and a Multi-Agent PPO (MAPPO) under identical budgets. Both approaches are based on the arm having $n$ number of controlled sections. The study systematically varies $n$ and evaluates the performance of the arm to reach a fixed target in three scenarios: default baseline condition, recovery from external disturbance, and adaptation to actuator failure. Quantitative metrics used for the evaluation are mean action magnitude, mean final distance, mean episode length, and success rate. The results show that there are no significant benefits of the distributed policy when the number of controlled sections $n\le4$. In very simple systems, when $n\le2$, the centralised policy outperforms the distributed one. When $n$ increases to $4< n\le 12$, the distributed policy shows a high sample efficiency. In these systems, distributed policy promotes a stronger success rate, resilience, and robustness under local observability and yields faster convergence given the same sample size. However, centralised policies achieve much higher time efficiency during training as it takes much less time to train the same size of samples. These findings highlight the trade-offs between centralised and distributed policy in reinforcement learning-based control for soft robotic systems and provide actionable design guidance for future sim-to-real transfer in soft rod-like manipulators.

Figures (4)

• Figure 1: (a) n-agent distributed control architecture: the soft arm is partitioned into n agents; each actor issues an individual force command at its section.Message passing supports coordination. A central critic evaluates value from the global state (CTDE). (b) Individual agent block: a communication module (encoder–aggregator) forms a learned representation from other agents’ observations. It is fused with the local state and passed to the policy network to produce the agent’s action.
• Figure 2: (a) Centralized policy: a single PPO controller receives global observations from the full rod and outputs joint actions to all forcing points. (b) Distributed policy: MAPPO with per-section agents. Each agent uses local observations message aggregation) to produce its own action, coordinating through a centralized critic under CTDE.
• Figure 3: (a) Sequential snapshots of the simulated soft robotic arm (blue) performing a fixed-point reaching task, where the rod tip $p^{\text{tip}}_t$ moves progressively toward the target $g$ (red star). (b) Illustration of the three testing scenarios used in the experiments: (left) baseline reaching under normal conditions; (middle) recovery from an externally applied disturbance $F$ to the mid-span rod; and (right) adaptation to actuator failure, where one or more sections (red) are disabled during execution.
• Figure 4: Training metrics of centralised (a) and distributed (b) policies over 10,000 updates for varying numbers of controlled sections $n$; shaded regions indicate variability across runs.