Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teaming

Giorgio Angelotti, Caroline P. C. Chanel, Adam H. M. Pinto, Christophe Lounis, Corentin Chauffaut, Nicolas Drougard

TL;DR

The paper tackles offline risk-sensitive decision-making in human-robot teaming under partial observability by learning a data-driven POMDP with Bayesian uncertainty from a fixed Firefighter Robot Game dataset. It extends Exploitation vs Caution (EvC) policy selection to POMDPs, sampling observation models from Dirichlet posteriors and evaluating policies via $VaR_{q}$ with $q=0.5$ across multiple discount factors using SARSOP. A practical pipeline trains four Extra Tree classifiers to map multimodal physiological and behavioral features to observations, defines a trivial POMDP, and computes robust policies that generalize across diverse participants. Validation with 26 participants shows the robust POMDP policy achieves higher scores than the data-collection policy, indicating improved robustness and generalization in HITL-RL under uncertainty. The work demonstrates how partial observability and model uncertainty can be addressed in offline HITL contexts, enabling safer, more effective human-robot collaboration in real-world tasks.

Abstract

The integration of physiological computing into mixed-initiative human-robot interaction systems offers valuable advantages in autonomous task allocation by incorporating real-time features as human state observations into the decision-making system. This approach may alleviate the cognitive load on human operators by intelligently allocating mission tasks between agents. Nevertheless, accommodating a diverse pool of human participants with varying physiological and behavioral measurements presents a substantial challenge. To address this, resorting to a probabilistic framework becomes necessary, given the inherent uncertainty and partial observability on the human's state. Recent research suggests to learn a Partially Observable Markov Decision Process (POMDP) model from a data set of previously collected experiences that can be solved using Offline Reinforcement Learning (ORL) methods. In the present work, we not only highlight the potential of partially observable representations and physiological measurements to improve human operator state estimation and performance, but also enhance the overall mission effectiveness of a human-robot team. Importantly, as the fixed data set may not contain enough information to fully represent complex stochastic processes, we propose a method to incorporate model uncertainty, thus enabling risk-sensitive sequential decision-making. Experiments were conducted with a group of twenty-six human participants within a simulated robot teleoperation environment, yielding empirical evidence of the method's efficacy. The obtained adaptive task allocation policy led to statistically significant higher scores than the one that was used to collect the data set, allowing for generalization across diverse participants also taking into account risk-sensitive metrics.

Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teaming

TL;DR

The paper tackles offline risk-sensitive decision-making in human-robot teaming under partial observability by learning a data-driven POMDP with Bayesian uncertainty from a fixed Firefighter Robot Game dataset. It extends Exploitation vs Caution (EvC) policy selection to POMDPs, sampling observation models from Dirichlet posteriors and evaluating policies via with across multiple discount factors using SARSOP. A practical pipeline trains four Extra Tree classifiers to map multimodal physiological and behavioral features to observations, defines a trivial POMDP, and computes robust policies that generalize across diverse participants. Validation with 26 participants shows the robust POMDP policy achieves higher scores than the data-collection policy, indicating improved robustness and generalization in HITL-RL under uncertainty. The work demonstrates how partial observability and model uncertainty can be addressed in offline HITL contexts, enabling safer, more effective human-robot collaboration in real-world tasks.

Abstract

The integration of physiological computing into mixed-initiative human-robot interaction systems offers valuable advantages in autonomous task allocation by incorporating real-time features as human state observations into the decision-making system. This approach may alleviate the cognitive load on human operators by intelligently allocating mission tasks between agents. Nevertheless, accommodating a diverse pool of human participants with varying physiological and behavioral measurements presents a substantial challenge. To address this, resorting to a probabilistic framework becomes necessary, given the inherent uncertainty and partial observability on the human's state. Recent research suggests to learn a Partially Observable Markov Decision Process (POMDP) model from a data set of previously collected experiences that can be solved using Offline Reinforcement Learning (ORL) methods. In the present work, we not only highlight the potential of partially observable representations and physiological measurements to improve human operator state estimation and performance, but also enhance the overall mission effectiveness of a human-robot team. Importantly, as the fixed data set may not contain enough information to fully represent complex stochastic processes, we propose a method to incorporate model uncertainty, thus enabling risk-sensitive sequential decision-making. Experiments were conducted with a group of twenty-six human participants within a simulated robot teleoperation environment, yielding empirical evidence of the method's efficacy. The obtained adaptive task allocation policy led to statistically significant higher scores than the one that was used to collect the data set, allowing for generalization across diverse participants also taking into account risk-sensitive metrics.
Paper Structure (24 sections, 5 figures, 1 table, 1 algorithm)

This paper contains 24 sections, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related works
  3. (PO)MDP Framework
  4. Offline POMDP model learning and solving
  5. The Firefighter Robot Game
  6. Brief environment description
  7. Data collection
  8. Risk-sensitive ORL with partial observability
  9. Data split.
  10. Classifier training
  11. Observation Function and Bayesian prior
  12. POMDP model definition
  13. Robust POMDP solving.
  14. Validation experiments
  15. Protocol
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Illustration of the proposed methodology.
  • Figure 2: Example of how the batch is split.
  • Figure 3: Computation of POMDP's marginalized belief of performance $\beta_t$ along a mission run using the POMDP policy. The control system switches the autonomy mode of the robot from manual to automatic (auto), and vice-versa. Notice how the belief is different from the immediate observation, as multiple subsequent non-performant observations can be necessary for the system to deem the state necessary to put the robot in automatic mode, e.g. from $t \approx 100 s$ to $t \approx 150 s$.
  • Figure 4: Scatter plot displaying the relationship between the average belief of human-robot performance during a mission (x-axis) and the mission score (y-axis). A fifth-order (monotonic) polynomial is fitted to show the positive correlation between $\overline{\beta}$ and the score. The shaded area represents plus or minus one standard deviation of the residuals of the fit.
  • Figure 5: Rest HRV per subject before a mission. Since these values are used to normalize the features to be used by the ML model, variability could lead to a failure of the pipeline.