Harnessing Data from Clustered LQR Systems: Personalized and Collaborative Policy Optimization

TL;DR

The paper tackles data inefficiency in model-free control by enabling personalized policies across heterogeneous yet related LQR systems. It introduces PCPO, a model-free, cluster-aware policy optimization framework that jointly identifies cluster structure via sequential elimination and learns per-cluster controllers using zeroth-order gradient estimates with a central aggregator. The authors prove that clustering is achieved with high probability and that, once clusters are correctly identified, collaboration within clusters yields a suboptimality gap that scales as $O\bigl(1/\sqrt{T|\mathcal{M}_{\sigma(i)}|}\bigr)$ without heterogeneity-induced bias, while requiring only logarithmic communication. The results demonstrate data-efficiency and stability when transferring information inside clusters and avoiding cross-cluster negative transfer, with practical distributed implementation benefits. This work lays groundwork for extending clustering-based data sharing to broader dynamical-control and RL settings.

Abstract

It is known that reinforcement learning (RL) is data-hungry. To improve sample-efficiency of RL, it has been proposed that the learning algorithm utilize data from 'approximately similar' processes. However, since the process models are unknown, identifying which other processes are similar poses a challenge. In this work, we study this problem in the context of the benchmark Linear Quadratic Regulator (LQR) setting. Specifically, we consider a setting with multiple agents, each corresponding to a copy of a linear process to be controlled. The agents' local processes can be partitioned into clusters based on similarities in dynamics and tasks. Combining ideas from sequential elimination and zeroth-order policy optimization, we propose a new algorithm that performs simultaneous clustering and learning to output a personalized policy (controller) for each cluster. Under a suitable notion of cluster separation that captures differences in closed-loop performance across systems, we prove that our approach guarantees correct clustering with high probability. Furthermore, we show that the sub-optimality gap of the policy learned for each cluster scales inversely with the size of the cluster, with no additional bias, unlike in prior works on collaborative learning-based control. Our work is the first to reveal how clustering can be used in data-driven control to learn personalized policies that enjoy statistical gains from collaboration but do not suffer sub-optimality due to inclusion of data from dissimilar processes. From a distributed implementation perspective, our method is attractive as it incurs only a mild logarithmic communication overhead.

Figures (1)

• Figure 1: Illustration of the epoch-based structure of PCPO, where each epoch involves three key steps: local policy optimization (PO), cost estimation, and global PO.

Theorems & Definitions (11)

• theorem 1
• theorem 2
• corollary 1
• lemma 1
• lemma 2
• lemma 3
• lemma 4
• proof
• corollary 2
• proof