Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints

TL;DR

Dual-Force tackles offline diversity maximization under imitation constraints by introducing a discriminator-free, Van der Waals based diversity objective combined with successor features. The method leverages Fenchel duality within the DICE framework for offline evaluation and learns policies conditioned on a pre-trained Functional Reward Encoding to handle non-stationary rewards and enable zero-shot recall of learned skills. This yields a stable offline training process and expands the set of learnable skills without explicit skill discrimination. Empirical results on SOLO12 locomotion and obstacle navigation demonstrate robust recovery of diverse behaviors and modalities from offline data, including multiple base-height modes and obstacle traversal strategies.

Abstract

While many algorithms for diversity maximization under imitation constraints are online in nature, many applications require offline algorithms without environment interactions. Tackling this problem in the offline setting, however, presents significant challenges that require non-trivial, multi-stage optimization processes with non-stationary rewards. In this work, we present a novel offline algorithm that enhances diversity using an objective based on Van der Waals (VdW) force and successor features, and eliminates the need to learn a previously used skill discriminator. Moreover, by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE), our method allows for better handling of non-stationary rewards and provides zero-shot recall of all skills encountered during training, significantly expanding the set of skills learned in prior work. Consequently, our algorithm benefits from receiving a consistently strong diversity signal (VdW), and enjoys more stable and efficient training. We demonstrate the effectiveness of our method in generating diverse skills for two robotic tasks in simulation: locomotion of a quadruped and local navigation with obstacle traversal.

Figures (6)

• Figure 1: Illustration of Dual-Force. The pseudocode is presented in \ref{['alg:dual-force']}.
• Figure 2: A performance benchmark of skills learned in the locomotion task. The SMODICE-expert walks with constant base and angular velocity, and with middle base-height. The learned skills recovers all base-height movements ( low, middle, high) and have different angular velocity. The colored horizontal bar at the top of each skill plot indicates the corresponding average base-height.
• Figure 3: Tasks: (a,b) Locomotion and (c,d) Navigation. The blue triangle is the SMODICE-expert and the colored dots are the learned skills. (a,c) Successor features pair-wise $\ell_2$ distances between skills. The first row (column) is SMODICE-expert, and all other rows (columns) are learned skills. (b,d) UMAP projection of successor features into 2D space.
• Figure 4: A performance benchmark of skills learned in the obstacle navigation task, where the SMODICE-expert is initialized in front of a box of height 0.2m and reaches a target position behind the box. The learned skills exhibit diverse behaviors that cover various modalities of expert and offline datasets.
• Figure 5: A performance benchmark with additional fence obstacles, of height 0.6m, partially blocking the path from (a) the left side, (b) the right side, or (c) both the left and right sides. Among the diverse skills learned, there are several that outperform the SMODICE-expert in (a,b) and perform on par with the SMODICE-expert in (c).

Theorems & Definitions (16)

• Lemma A.2: State-only KL Estimator
• proof
• Corollary A.3: Structural
• proof
• Claim A.4
• proof
• Claim A.5
• proof
• Claim A.6
• proof