Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing

TL;DR

It is shown that the combined human-robot team, when blending its actions with those of the human, outperforms the synthetic humans alone as well as several baseline assistance strategies, and that intent-conditioning enables adherence to human preferences during task execution, leading to improved performance while satisfying the human's objective.

Abstract

Tight coordination is required for effective human-robot teams in domains involving fast dynamics and tactical decisions, such as multi-car racing. In such settings, robot teammates must react to cues of a human teammate's tactical objective to assist in a way that is consistent with the objective (e.g., navigating left or right around an obstacle). To address this challenge, we present Dream2Assist, a framework that combines a rich world model able to infer human objectives and value functions, and an assistive agent that provides appropriate expert assistance to a given human teammate. Our approach builds on a recurrent state space model to explicitly infer human intents, enabling the assistive agent to select actions that align with the human and enabling a fluid teaming interaction. We demonstrate our approach in a high-speed racing domain with a population of synthetic human drivers pursuing mutually exclusive objectives, such as "stay-behind" and "overtake". We show that the combined human-robot team, when blending its actions with those of the human, outperforms the synthetic humans alone as well as several baseline assistance strategies, and that intent-conditioning enables adherence to human preferences during task execution, leading to improved performance while satisfying the human's objective.

Figures (26)

• Figure 1: We mimic human preferences on discrete decisions via population clusters during world model formation. Our assistive models then learn to decisively help on the overall discrete-continuous control task while taking into account multiple possible human preferences.
• Figure 2: Overview of value alignment for the assistive agent. We start with a set of frozen human policies whose values are annotated with predetermined outcomes. We blend human actions linearly with the actions from an unfrozen assistive agent. For both human and assistive agents, a multi-head RSSM architecture is then used to predict the observations, reward, and human intent (assistive only), trained to maximize their log-likelihoods against samples taken from the environment. The intent head is trained to match frozen human intents, which are fixed a priori. The assistive agent's rewards are shaped based on the optimal policy$\pi^*_{\hat{y}^A}$ and optimal predicted reward $r^*_{\hat{y}^A}$ for inferred intent $\hat{y}^A$.
• Figure 3: Examples of the Dream2Assist agent's actions when paired with a human intending to pass and a human intending to stay. Dream2Assist recognizes the driver's intent, making lateral corrections for a safer overtake (left) or throttle adjustments to stay behind the opponent while still progressing towards the finish (right), thereby helping to satisfy task and human objectives.
• Figure B.1: Consistency of overtake versus non-overtakes.
• Figure B.2: Consistency of left-handed versus right-handed overtakes.