To Ask or Not To Ask: Human-in-the-loop Contextual Bandits with Applications in Robot-Assisted Feeding

TL;DR

This work develops a human-in-the-loop contextual bandit framework for robot-assisted bite acquisition, introducing LinUCB-QG, which balances autonomous action with selective user querying by leveraging a data-driven workload predictor. The workload model is trained on a diverse dataset including mobility-impaired users and OTs, enabling the system to anticipate querying workload and adapt its querying strategy. In simulation and a real-world Kinova-based study with 19 participants, LinUCB-QG achieves higher task success than autonomous baselines while imposing substantially less workload than always querying, including significant gains for users with mobility limitations. The approach advances practical, user-centered robotic feeding by reducing cognitive and physical burden through nonintrusive workload modeling and adaptive querying policies with strong empirical support. Collectively, the work demonstrates the feasibility and value of workload-aware human-in-the-loop policies for complex manipulation tasks in assistive robotics.

Abstract

Robot-assisted bite acquisition involves picking up food items with varying shapes, compliance, sizes, and textures. Fully autonomous strategies may not generalize efficiently across this diversity. We propose leveraging feedback from the care recipient when encountering novel food items. However, frequent queries impose a workload on the user. We formulate human-in-the-loop bite acquisition within a contextual bandit framework and introduce LinUCB-QG, a method that selectively asks for help using a predictive model of querying workload based on query types and timings. This model is trained on data collected in an online study involving 14 participants with mobility limitations, 3 occupational therapists simulating physical limitations, and 89 participants without limitations. We demonstrate that our method better balances task performance and querying workload compared to autonomous and always-querying baselines and adjusts its querying behavior to account for higher workload in users with mobility limitations. We validate this through experiments in a simulated food dataset and a user study with 19 participants, including one with severe mobility limitations. Please check out our project website at: http://emprise.cs.cornell.edu/hilbiteacquisition/

Figures (5)

• Figure 1: We present a human-in-the-loop contextual bandit-based framework for robot-assisted bite acquisition and propose a novel method which decides whether to ask for help or autonomously act, by considering both certainty about action performance, and the estimated workload imposed on the human by querying.
• Figure 2: Human-in-the-loop contextual bandit pipeline for deciding whether to query ($a_q$) or autonomously select a robot action $a_r$. Our proposed method, LinUCB-QG, takes into account action uncertainty as measured by the performance gap $G$ between the best and the second best robot action and incorporates a learned querying workload model ($f$) to predict the workload of querying the human $WL_{t}$.
• Figure 3: Left: Querying workload user study setup, illustrating the format of the distraction task and robot query task, and the modified NASA-TLX survey recorded after every robot query. Right: Independent variables that affect user's querying workload varied in the study.
• Figure 4: Simulation and real-world results. Left (bottom): Workload comparison between three querying algorithms in simulation ($D_2$ setting), with a sequence of 5 food items. Vertical lines indicate start of new food item, while red circles indicate query timesteps. LinUCB-QG reduces querying workload $WL_t$ compared to the other two methods, while offering competitive convergence times compared to LinUCB-ExpDecay. Left (top):LinUCB-QG balances real-world performance and user workload, significantly reducing workload compared to Always-Query (which achieves slightly higher success at greater user cost), as shown by three post-method metrics (Post) and three change in workload metrics (Delta). Right:LinUCB-QG significantly outperforms LinUCB, our autonomous baseline, in real-world bite acquisition success, both subjectively (Post-Performance) and objectively ($r_{task,avg}$). Error bars indicate standard error.
• Figure 5: Example LinUCB-QG rollout in simulation environment, illustrating evolution of performance gap $G$ and workload $WL_t$, along with timesteps corresponding to query times. LinUCB-QG decides to ask the human for help when $G > w WL_t$ (in this rollout, $w = 4$).