Imitation Learning via Focused Satisficing

TL;DR

The paper reframes imitation learning through satisficing theory, arguing that demonstrators often prioritize acceptable rather than optimal behavior and introduces Minimally Subdominant Focused Imitation (MinSubFI) that minimizes subdominance to surpass aspirational levels on unseen demonstrations. It develops a margin-based, snippet- and stochastic-policy-enabled objective, derives policy-gradient updates, and provides online/offline learning variants plus a learned-cost-feature representation from pairwise preferences. The approach yields higher demonstrator acceptability and competitive true returns across CartPole, LunarLander, Hopper, HalfCheetah, and Walker, including robustness to suboptimal and human demonstrations. This work offers a practical imitation-learning paradigm with explicit acceptability guarantees that can adapt to dynamic aspirational criteria without requiring explicit reward reconstruction. The framework’s generalization bounds and snippet-focused mechanisms further enhance reliability in real-world, varying-aspiration settings.

Abstract

Imitation learning often assumes that demonstrations are close to optimal according to some fixed, but unknown, cost function. However, according to satisficing theory, humans often choose acceptable behavior based on their personal (and potentially dynamic) levels of aspiration, rather than achieving (near-) optimality. For example, a lunar lander demonstration that successfully lands without crashing might be acceptable to a novice despite being slow or jerky. Using a margin-based objective to guide deep reinforcement learning, our focused satisficing approach to imitation learning seeks a policy that surpasses the demonstrator's aspiration levels -- defined over trajectories or portions of trajectories -- on unseen demonstrations without explicitly learning those aspirations. We show experimentally that this focuses the policy to imitate the highest quality (portions of) demonstrations better than existing imitation learning methods, providing much higher rates of guaranteed acceptability to the demonstrator, and competitive true returns on a range of environments.

Figures (6)

• Figure 1: Left: Pareto-dominating in the cost function bases (${\textrm{f}}_1$, ${\textrm{f}}_2$) of acceptable behavior (purple: imitator acceptable set) guarantees the imitator is acceptable to the demonstrator (red: demonstrator acceptable set). Right: The subdominance (orange lines) measures how far imitator trajectory rollouts are from guaranteed acceptance (by a margin).
• Figure 2: Existing reward-based imitation methods, e.g., TREX brown2019extrapolating, seek to outperform demonstrations using a pipeline of engineered components (top) to first segment trajectories into "snippets," and to ultimately estimate a reward function that is then optimized using reinforcement learning. Our approach (bottom) uses the subdominance as the reinforcement learning objective, which is defined by the relative performance of the imitator compared to the demonstrations in each cost feature. This effectively uses feedback from the learned imitator policy to guide additional reinforcement learning without an explicit reward function.
• Figure 3: Examples of lunarlander cost features, which are computed easily from the environment's observation vector.
• Figure 4: Mean true returns of 100 trajectories rolled out from the learned policies and the minimum, average, and maximum reward of the training set trajectories . Each policy was trained on a subset of demonstrations obtained by removing the best or worst $10\%$, $20\%$, $30\%$, or $40\%$ of the demonstrations. Compared to T-REX (orange) and AIRL (purple), the performance of MinSubFI (green) is more robust to increases in the proportion of suboptimal demonstrations in the dataset.
• Figure 5: Implemented architecture of Online MinSubFI (top) and Offline MinSubFI (bottom) using $\texttt{gym}$ and $\texttt{stable-baselines3}$. The primary functionality of cost feature and subdominance computation is tackled via two environment wrappers $\texttt{CostFeatureWrapper}$ and $\texttt{SubdominanceRewardWrapper}$. The former computes cost features from observations and the latter computes subdominance relative to demonstrations using cost features. $\texttt{SubdominanceCallback}$ is called periodically to update $\alpha$. Offline MinSubFI uses the $\texttt{OfflineDemoInjector}$ wrapper around an environment to pass $(s,a,r,s',f)$ tuples from demonstrations as rollout data instead, and there is no action returned from the policy to the environment.

Theorems & Definitions (16)

• Definition 1
• Theorem 2
• proof : Proof of Theorem \ref{['thm:satisficing']}
• Definition 3
• Theorem 4
• Corollary 5
• Corollary 6
• Definition 7
• Theorem 8
• Definition 9