Human-in-the-Loop Failure Recovery with Adaptive Task Allocation

TL;DR

The paper tackles failures in autonomous robots operating in unstructured settings by introducing ARFA, an adaptive, real-time failure allocation framework that assigns failures to local or remote operators to optimize recovery outcomes. ARFA models operator capabilities in three dimensions $C=\{phys, cog, resp\}$ and maintains online beliefs $bel(c_{i,j})=(\ell_{i,j}, u_{i,j})$, selecting the operator that maximizes the expected reward $\text{ER}_i=\Lambda_i( R_i - C_i)$, with $\Lambda_i$ capturing capability fit and $R_i$ and $C_i$ encoding urgency and workload. The framework updates capabilities online via an Adam-based optimizer (learning rate $0.001$, decay $0.99$) by minimizing the loss between observed and predicted performance, where $\hat{\mathcal{P}}_i(\mathbf{r})=\prod_{j}\lambda_{i,j}(r_j)$ and $\mathcal{P}_i(\mathbf{r})=S_i(\mathbf{r})/T_i(\mathbf{r})$. Validation through simulations and a user study with the IONA nursing robot shows ARFA reduces robot idle time, increases team and local-user performance, and achieves a more balanced workload, demonstrating practical benefits for real-time human-robot collaboration in healthcare and service domains.

Abstract

Since the recent Covid-19 pandemic, mobile manipulators and humanoid assistive robots with higher levels of autonomy have increasingly been adopted for patient care and living assistance. Despite advancements in autonomy, these robots often struggle to perform reliably in dynamic and unstructured environments and require human intervention to recover from failures. Effective human-robot collaboration is essential to enable robots to receive assistance from the most competent operator, in order to reduce their workload and minimize disruptions in task execution. In this paper, we propose an adaptive method for allocating robotic failures to human operators (ARFA). Our proposed approach models the capabilities of human operators, and continuously updates these beliefs based on their actual performance for failure recovery. For every failure to be resolved, a reward function calculates expected outcomes based on operator capabilities and historical data, task urgency, and current workload distribution. The failure is then assigned to the operator with the highest expected reward. Our simulations and user studies show that ARFA outperforms random allocation, significantly reducing robot idle time, improving overall system performance, and leading to a more distributed workload among operators.

Figures (5)

• Figure 1: Our Adaptive Robotic Failure Allocation (ARFA) method allocates failures to the most suitable operator given the failure's requirements, the operators' capabilities, their historical performance in failure recovery, and workload distribution.
• Figure 2: For a given failure, the performance index for each operator is calculated given the failure requirements and current capabilities beliefs. The failure is then allocated to the operator with the highest expected reward. Post-resolution, data is collected, and the operator's capabilities are updated based on the failure recovery outcome.
• Figure 3: Simulation result of one representative trial of the remote operator. For each failure assigned to this operator, the plots show the respective requirements for cognitive ability, physical ability, and urgency. Successfully resolved failures are marked with blue circles, while red crosses indicate unsuccessful ones. Upper and lower bounds of the operator's beliefs for each capability are also shown.
• Figure 4: Experiment Setup: on the left, the remote operator uses a screen-based graphical user interface. On the right, the robotic system and the local operator wearing Microsoft HoloLens 2.
• Figure 5: User study results. (a) Total task completion time was significantly reduced with ARFA allocation. (b) The team success rate, which is the ability to resolve a failure within a predefined time threshold based on the failure urgency, significantly increased with ARFA. (c) ARFA distributes the workload between operators, measured as the total failure recovery duration for each operator.