Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration

TL;DR

The paper tackles how to initialize assistive agents with a policy that can quickly adapt to a partner's evolving reward function. It introduces BLR-HAC, a two-stage method that bootstraps a large nonlinear model to initialize a lightweight online logistic-regression-based policy, enabling fast, in-task adaptation. Through simulated surface rearrangement tasks, BLR-HAC delivers strong zero-shot performance with inductive priors and substantially lowers online compute while matching the performance of fine-tuned large models. This approach promises practical benefits for real-time human-agent collaboration, balancing effective initialization with rapid personalization.

Abstract

Agents that assist people need to have well-initialized policies that can adapt quickly to align with their partners' reward functions. Initializing policies to maximize performance with unknown partners can be achieved by bootstrapping nonlinear models using imitation learning over large, offline datasets. Such policies can require prohibitive computation to fine-tune in-situ and therefore may miss critical run-time information about a partner's reward function as expressed through their immediate behavior. In contrast, online logistic regression using low-capacity models performs rapid inference and fine-tuning updates and thus can make effective use of immediate in-task behavior for reward function alignment. However, these low-capacity models cannot be bootstrapped as effectively by offline datasets and thus have poor initializations. We propose BLR-HAC, Bootstrapped Logistic Regression for Human Agent Collaboration, which bootstraps large nonlinear models to learn the parameters of a low-capacity model which then uses online logistic regression for updates during collaboration. We test BLR-HAC in a simulated surface rearrangement task and demonstrate that it achieves higher zero-shot accuracy than shallow methods and takes far less computation to adapt online while still achieving similar performance to fine-tuned, large nonlinear models. For code, please see our project page https://sites.google.com/view/blr-hac.

Figures (4)

• Figure 1: One step of an example surface rearrangement task: cupboard organization. From left to right: a person (H) picks an object to place in the dishwasher; the agent (A) initially places this incorrectly; the person corrects the placement. From this, the agent learns that the user likes to place blue objects on the bottom shelf and can place the next, similar object correctly.
• Figure 2: BLR-HAC Overview From left to right, we first embed the input state and actions using $\phi$. These are then concatenated and fed into the preference estimator $\psi$. This learns to output reward parameters, $\theta$ which are used to initialize an online learning policy using the policy $\pi$, which determines the robot's action $a_r$.
• Figure 3: Stationary Test-Time Adaptation. Learning curves for each test environment for each algorithm. We report the average accuracy over each episode. BLR-HAC is able to achieve the low zero-shot performance of the transformer method, and the fast adaptation of the linear method. Additionally, we can see that as the episode length increases, these differences in performance are more notable, with the linear method failing to catch up to the other two methods over the course of 20 episodes.
• Figure 4: Nonstationary Test-Time Adaptation. Learning curves over each test environment for each algorithm. We report average accuracy over episodes. BLR-HAC is able to perform on par with the transformer method in the small and medium environments and part of the large environment. BLR-HAC outperforms all methods in all environments immediately after the preference switch. In the large environment, though, the transformer recovers more quickly as it has access to more data.