End-to-end Optimization of Belief and Policy Learning in Shared Autonomy Paradigms

TL;DR

BRACE tackles the challenge of inferring user intent under goal uncertainty and environmental constraints in shared autonomy by end-to-end optimizing a belief-conditioned policy for adaptive assistance. It introduces a dual-head architecture with a Bayesian inference module and a context-aware PPO-based arbiter, where the continuous blending parameter $\gamma$ is conditioned on the full belief distribution $b$ and context $c$, enabling gradients to shape both belief updates and control. Theoretical results establish that optimal assistance decreases with belief entropy and increases with constraint severity, and that integrated belief conditioning yields a quadratic regret advantage over MAP-based approaches. Empirically, BRACE outperforms state-of-the-art baselines (IDA, DQN) across planar cursor control, Reacher-2D, and FetchPickAndPlace tasks, with particularly strong gains in high-ambiguity, constrained scenarios, demonstrating robust, generalizable improvements for adaptive shared autonomy.

Abstract

Shared autonomy systems require principled methods for inferring user intent and determining appropriate assistance levels. This is a central challenge in human-robot interaction, where systems must be successful while being mindful of user agency. Previous approaches relied on static blending ratios or separated goal inference from assistance arbitration, leading to suboptimal performance in unstructured environments. We introduce BRACE (Bayesian Reinforcement Assistance with Context Encoding), a novel framework that fine-tunes Bayesian intent inference and context-adaptive assistance through an architecture enabling end-to-end gradient flow between intent inference and assistance arbitration. Our pipeline conditions collaborative control policies on environmental context and complete goal probability distributions. We provide analysis showing (1) optimal assistance levels should decrease with goal uncertainty and increase with environmental constraint severity, and (2) integrating belief information into policy learning yields a quadratic expected regret advantage over sequential approaches. We validated our algorithm against SOTA methods (IDA, DQN) using a three-part evaluation progressively isolating distinct challenges of end-effector control: (1) core human-interaction dynamics in a 2D human-in-the-loop cursor task, (2) non-linear dynamics of a robotic arm, and (3) integrated manipulation under goal ambiguity and environmental constraints. We demonstrate improvements over SOTA, achieving 6.3% higher success rates and 41% increased path efficiency, and 36.3% success rate and 87% path efficiency improvement over unassisted control. Our results confirmed that integrated optimization is most beneficial in complex, goal-ambiguous scenarios, and is generalizable across robotic domains requiring goal-directed assistance, advancing the SOTA for adaptive shared autonomy.

Figures (13)

• Figure 1: A general formulation of our shared autonomy framework: adaptive blending of human kinematic input ($K_{user}$) and expert policy's kinematic input ($K_{expert}$) via $\gamma$ in Shared Autonomy frameworks.
• Figure 2: Architectural overview of our dual-head shared autonomy system. The network integrates a Bayesian goal inference module with a context-adaptive assistance policy learned through actor-critic reinforcement learning. The Bayesian inference module calculates belief states which are passed to the policy module to condition assistance decisions.
• Figure 3: Architecture of the assistance arbitration PPO. The policy module consists of an Actor Head generating the blending parameter $\gamma$ and a Critic Head estimating state value. Both share a Feature Fusion Block that processes combined state and belief information.
• Figure 4: Structure of the human-in-the-loop cursor control user study, showing the distribution of trials across three test environments and five assistance conditions.
• Figure 5: Comparison between manual gamma (left) and BRACE (right). Color scale shows assistance level from 0 (purple, human control) to 1 (red, AI control). Green circles are targets, gray circles are obstacles. BRACE dynamically adjusts assistance based on context and goal certainty.

Theorems & Definitions (2)

• Theorem 1: Uncertainty and constraint monotonicity
• Theorem 2: Integrated optimization advantage