Causal Coordinated Concurrent Reinforcement Learning

TL;DR

This work tackles data-efficient policy learning in concurrent RL when agents operate in non-identical MDPs by using Additive Noise Model - Mixture Model (ANM-MM) to extract latent environment-differentiating parameters. A soft-mechanism clustering approach yields similarity-based data sharing, while a simple seed-based coordinated exploration heuristic promotes diverse state visitation under sparse rewards. Empirical results across autoregressive, pendulum, and cart-pole tasks demonstrate faster learning and robust performance when leveraging latent-parameter clustering and coordinated action selection, outperforming baselines that assume identical environments or ignore coordination. The study advances CRL by integrating causal inference with RL and connects to transportability, offering a principled way to transfer policies across related but varied environments.

Abstract

In this work, we propose a novel algorithmic framework for data sharing and coordinated exploration for the purpose of learning more data-efficient and better performing policies under a concurrent reinforcement learning (CRL) setting. In contrast to other work which make the assumption that all agents act under identical environments, we relax this restriction and instead consider the formulation where each agent acts within an environment which shares a global structure but also exhibits individual variations. Our algorithm leverages a causal inference algorithm in the form of Additive Noise Model - Mixture Model (ANM-MM) in extracting model parameters governing individual differentials via independence enforcement. We propose a new data sharing scheme based on a similarity measure of the extracted model parameters and demonstrate superior learning speeds on a set of autoregressive, pendulum and cart-pole swing-up tasks and finally, we show the effectiveness of diverse action selection between common agents under a sparse reward setting. To the best of our knowledge, this is the first work in considering non-identical environments in CRL and one of the few works which seek to integrate causal inference with reinforcement learning (RL).

Figures (6)

• Figure 1: An autoencoder interpretation of our model.
• Figure 2: A comparison between our model vs. three baselines for $s_*$ sampled from a trimodal GMM on the AR task. Shaded region represents one SD of uncertainty from 30 sampled environments.
• Figure 3: Histogram of the extracted model parameters fitted with GMM clustering for the (a) AR task, (b) AR task with sparse rewards, (c) windy pendulum task and (d) cart-pole swing-up task. Bars of the histogram are color-coded according to their true data generating mechanism.
• Figure 4: (a) and (b): A comparison between our coordinated action selection heuritistic vs. a CRL agnostic baseline for $s_*$ sampled from a bimodal GMM on the sparse reward AR task. Shaded region represents one SD of uncertainty from 18 sampled environments. (c)-(f): State trajectories of the AR processes for various instances.
• Figure 5: A comparison between our model vs. three baselines for wind strengths sampled from a multimodal GMM on the windy pendulum task. Shaded region represents one SD of uncertainty from 20 sampled environments.