Swarm Behavior Cloning

TL;DR

Swarm Behavior Cloning (Swarm BC) tackles the problem of divergent action predictions among ensemble BC policies in offline imitation learning. By adding a regularization term that encourages alignment of hidden feature representations across ensemble members, Swarm BC reduces action divergence while preserving diversity, leading to more reliable aggregated actions. Empirical evaluation on eight OpenAI Gym environments shows consistent performance gains and reduced mean action differences, particularly in higher-dimensional spaces. Theoretical analysis connects the training objective to concentrating probability mass on the global hidden-feature mode, offering a principled justification for the observed improvements and suggesting strong practical impact for offline imitation and ensemble methods.

Abstract

In sequential decision-making environments, the primary approaches for training agents are Reinforcement Learning (RL) and Imitation Learning (IL). Unlike RL, which relies on modeling a reward function, IL leverages expert demonstrations, where an expert policy $π_e$ (e.g., a human) provides the desired behavior. Formally, a dataset $D$ of state-action pairs is provided: $D = {(s, a = π_e(s))}$. A common technique within IL is Behavior Cloning (BC), where a policy $π(s) = a$ is learned through supervised learning on $D$. Further improvements can be achieved by using an ensemble of $N$ individually trained BC policies, denoted as $E = {π_i(s)}{1 \leq i \leq N}$. The ensemble's action $a$ for a given state $s$ is the aggregated output of the $N$ actions: $a = \frac{1}{N} \sum{i} π_i(s)$. This paper addresses the issue of increasing action differences -- the observation that discrepancies between the $N$ predicted actions grow in states that are underrepresented in the training data. Large action differences can result in suboptimal aggregated actions. To address this, we propose a method that fosters greater alignment among the policies while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. We evaluate our approach across eight diverse environments, demonstrating a notable decrease in action differences and significant improvements in overall performance, as measured by mean episode returns.

Figures (5)

• Figure 1: This figure visualizes schematically the predicted actions of three different Behavior Cloning approaches, represented as black dots, in a 2-dim action space for some state $s_t$. The heatmap represents the Q-values $Q(a_t, s_t)$. (Left) the left plot shows plain Behavior Cloning. A policy $\pi$ was trained using supervised learning on some training data $D$. The black dot is the predicted action $a_t = \pi(s_t)$. (Middle) in Ensemble Behavior Cloning an ensemble of $N$ policies is trained individually on $D$. The $N$ predicted actions $\{a_i = \pi_i(s_t)\}$ (gray dots) are then aggregated to the ensemble action (black dot). (Right) in our approach Swarm Behavior Cloning an ensemble of $N$ policies is trained as well. However, they are not trained individually but using a modified loss function, see formula (2). The effect is a smaller difference between the $N$ predicted actions, resembling a swarm behavior. Similar to Ensemble Behavior Cloning the ensemble action (black dot) is then aggregated from the $N$ predicted actions (gray dots).
• Figure 2: This figure visualizes exemplarily the mean action difference for an entire episode of an ensemble containing $N = 6$ policies. We used the LunarLander-continuous environment since it has a 2-dim action space that can be easily visualized. The x-axis in the left plot represents the timestep in the episode. For two interesting timesteps, we have visualized the predicted actions of the $N$ policies $\{a^i_t = \pi_i(s_t)\}$ (gray dots) as well as the aggregated action (black dot) on a 2-dim map (the complete action space). The underlying heatmap represents the Q-values from the expert critic (a fully-trained SAC model from Stable-Baselines 3).
• Figure 3: These plots show the mean normalized test returns of our approach Swarm BC and two baseline algorithms on eight different OpenAI Gym environments. The graphs represent the mean over 20 episodes and 5 seeds. The x-axes represent the number of expert episodes in the training data $D$. The results show a significant performance improvement in environments with larger observation- and action spaces.
• Figure 4: In this figure we are evaluating whether Swarm BC can reduce the mean action difference as defined in Definition 3.1, which is the difference between the $N$ predicted actions $\{a_i = \pi_i(s)\}_{1 \leq i \leq N}$ of an ensemble $E$ containing $N$ policies. The results show that our approach does indeed reduce it but depending on the environment not always to the same extent. The x-axes in these plots represent the timestep in the test episodes and the y-axes represent the mean action difference. The graphs are the mean over $20$ episodes and $5$ seeds.
• Figure 5: To examine the sensitivity of the two hyperparameters $\tau$ and $N$ we did an ablation study. (Left) choosing $\tau$ too large or too small can reduce the test performance in terms of mean episode return. For Walker2D the best value was $\tau = 0.25$. Thus, we chose this value for all experiments in this paper. (Right) the conclusion of the ablation on the ensemble size $N$ is that larger $N$ are better, but this comes at the expense of longer runtime. For $N > 4$, however, the performance does not increase significantly anymore. Thus we chose $N = 4$ for all experiments in this paper.

Theorems & Definitions (1)

• Definition 1: Mean Action Difference