Collaborating in a competitive world: Heterogeneous Multi-Agent Decision Making in Symbiotic Supply Chain Environments

TL;DR

This work studies heterogeneous versus homogeneous multi-agent reinforcement learning in a two-echelon symbiotic supply chain (factory and retailer) under high and low demand. It formalizes a heterogeneous hidden Markov decision process, compares PPO and SAC algorithms, and introduces reward shaping that penalizes other agents’ stockouts without profit sharing. Key findings show SAC generally outperforms PPO in high demand and that heterogeneous agents mitigate bullwhip effects, but in low demand homogeneous architectures often perform better due to observability and pricing dynamics. The results highlight the importance of demand regime and observability for multi-agent coordination in supply chains and suggest limited but context-dependent benefits from reward shaping in collaborative settings.

Abstract

Supply networks require collaboration in a competitive environment. To achieve this, nodes in the network often form symbiotic relationships as they can be adversely effected by the closure of companies in the network, especially where products are niche. However, balancing support for other nodes in the network against profit is challenging. Agents are increasingly being explored to define optimal strategies in these complex networks. However, to date much of the literature focuses on homogeneous agents where a single policy controls all of the nodes. This isn't realistic for many supply chains as this level of information sharing would require an exceptionally close relationship. This paper therefore compares the behaviour of this type of agent to a heterogeneous structure, where the agents each have separate polices, to solve the product ordering and pricing problem. An approach to reward sharing is developed that doesn't require sharing profit. The homogenous and heterogeneous agents exhibit different behaviours, with the homogenous retailer retaining high inventories and witnessing high levels of backlog while the heterogeneous agents show a typical order strategy. This leads to the heterogeneous agents mitigating the bullwhip effect whereas the homogenous agents do not. In the high demand environment, the agent architecture dominates performance with the Soft Actor-Critic (SAC) agents outperforming the Proximal Policy Optimisation (PPO) agents. Here, the factory controls the supply chain. In the low demand environment the homogenous agents outperform the heterogeneous agents. Control of the supply chain shifts significantly, with the retailer outperforming the factory by a significant margin.

Figures (18)

• Figure 1: Multi-agent approach to solving an inventory dynamics model in a two-echelon supply chain.
• Figure 2: Comparison of agent's performance in homogeneous and heterogeneous configurations, using PPO and SAC architectures in the high demand environment. The shaded area depicts the standard deviation of the multi-agent's performance for independent experiments using 5 different seeds.
• Figure 3: Comparison of the factory and the retailer agent's reward for the SAC and PPO architectures in the high demand environment. The shaded area depicts the standard deviation of the heterogeneous agent's performance for independent experiments using 5 different seeds.
• Figure 4: Heterogeneous SAC agent actions and resulting backlog and stockouts in the high demand scenario. The mean inventory of the retailer, on the left, is 11.1 while the mean inventory of the factory, on the right, is 28.
• Figure 5: Homogeneous SAC agent actions and resulting backlog and stockout in the high demand scenario. The inventory of the factory is 3.13 while the inventory of the retailer is 18.91.