Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluation

TL;DR

The paper tackles the challenge of deploying RL in human–robot collaboration by automating RL design choices through off‑policy evaluation (OPE) on logged data. It introduces a unified offline framework that simultaneously evaluates candidate state spaces and reward functions, training offline RL agents and ranking them via ${\hat{V}_{OPE}(\\pi)}$ to identify optimal configurations. The approach is validated in two environments: Lunar Lander and NASA MATB-II, demonstrating the framework's ability to select informative state representations (e.g., ${S_{orig}}$ vs ${S_{more}}$, ${S_{task+physio+wl}}$) and effective reward functions (e.g., ${f(s,a)}$, ${r_3(s,a)}$) without online interaction. Online experiments with human participants corroborate that the chosen offline configurations can yield high task performance and favorable user experience, though personalization (RLFT) may increase cognitive workload and reduce predictability under high load. Overall, the work advances safe, data‑driven RL formulation for complex human‑robot teaming by reducing reliance on real‑time interactions while highlighting tradeoffs between autonomy, trust, and usability.

Abstract

Reinforcement learning (RL) has the potential to transform real-world decision-making systems by enabling autonomous agents to learn from experience. Deploying RL in real-world settings, especially in the context of human-robot interaction, requires defining state representations and reward functions, which are critical for learning efficiency and policy performance. Traditional RL approaches often rely on domain expertise and trial-and-error, necessitating extensive human involvement as well as direct interaction with the environment, which can be costly and impractical, especially in complex and safety-critical applications. This work proposes a novel RL framework that leverages off-policy evaluation (OPE) for state space and reward function selection, using only logged interaction data. This approach eliminates the need for real-time access to the environment or human-in-the-loop feedback, greatly reducing the dependency on costly real-time interactions. The proposed approach systematically evaluates multiple candidate state representations and reward functions by training offline RL agents and applying OPE to estimate policy performance. The optimal state space and reward function are selected based on their ability to produce high-performing policies under OPE metrics. Our method is validated on two environments: the Lunar Lander environment by OpenAI Gym, which provides a controlled setting for assessing state space and reward function selection, and a NASA-MATB-II human subjects study environment, which evaluates the approach's real-world applicability to human-robot teaming scenarios. This work enhances the feasibility and scalability of offline RL for real-world environments by automating critical RL design decisions through a data-driven OPE-based evaluation, enabling more reliable, effective, and sustainable RL formulation for complex human-robot interaction settings.

Figures (10)

• Figure 1: The proposed scheme for state space selection using off-policy evaluation (OPE).
• Figure 2: Our process for selecting a reward function based on the separability of good and bad policies using OPE.
• Figure 3: Lunar Lander Environment.
• Figure 4: Average rewards over $1000$ episodes for a DDQN agent trained on the Lunar Lander environment. The model, at episode $100$, is used as the Avg. DDQN model, whereas the model at episode $607$ is used as the Best DDQN model.
• Figure 5: The NASA Multi-Attribute Task Battery-II (MATB-II) Environment.