CoMI-IRL: Contrastive Multi-Intention Inverse Reinforcement Learning

TL;DR

This work addresses multi-intention IRL by decoupling behavior clustering from reward learning. It introduces CoMI-IRL, a transformer-based unsupervised framework that learns trajectory embeddings through contrastive learning and discovers behavioral modes via graph connectivity without requiring the true number of modes $K^*$; reward learning is then performed independently per cluster. The approach enables adaptation to unseen behaviors through encoder finetuning and a two-stage clustering process, improving clustering quality and downstream reward performance on MuJoCo tasks. By visualizing embedding relationships and decoupling analysis from rewards, CoMI-IRL enhances interpretability and applicability of MI-IRL to dynamic, multi-behavior scenarios.

Abstract

Inverse Reinforcement Learning (IRL) seeks to infer reward functions from expert demonstrations. When demonstrations originate from multiple experts with different intentions, the problem is known as Multi-Intention IRL (MI-IRL). Recent deep generative MI-IRL approaches couple behavior clustering and reward learning, but typically require prior knowledge of the number of true behavioral modes $K^*$. This reliance on expert knowledge limits their adaptability to new behaviors, and only enables analysis related to the learned rewards, and not across the behavior modes used to train them. We propose Contrastive Multi-Intention IRL (CoMI-IRL), a transformer-based unsupervised framework that decouples behavior representation and clustering from downstream reward learning. Our experiments show that CoMI-IRL outperforms existing approaches without a priori knowledge of $K^*$ or labels, while allowing for visual interpretation of behavior relationships and adaptation to unseen behavior without full retraining.

Figures (6)

• Figure 1: CoMI-IRL pipeline. States and actions of each quantile normalized input trajectory $\tau_i$ pass through a Random Fourier Feature (RFF) encoding with Gaussian mapping, a shallow MLP and a 1D CNN layer. After adding timestep and modality embeddings (distinguishing states and actions), we interleave the two sequences and prepend a $CLS$ token summarizing the trajectory. On the resulting embeddings, we apply graph-clustering to get cluster labels and apply IRL independently on each cluster.
• Figure 2: 2D UMAP visualizations of the original trajectory space of each environment (from top to bottom: Reacher, Pusher, Walker2D), and the respective embedding space resulting from the BE.
• Figure 3: Left to Right: Reacher, Pusher, Walker-2D.
• Figure 4: Jacobian Edge Reweighting on Pusher.
• Figure 5: Jacobian Edge Reweighting on Reacher.