Reinforcement Learning from Implicit Neural Feedback for Human-Aligned Robot Control

TL;DR

The paper tackles the challenge of learning effective robotic policies under sparse rewards by introducing Reinforcement Learning from Implicit Human Feedback (RLIHF), which decodes error-related potentials (ErrPs) from EEG into continuous reward signals. The ErrP-based rewards are integrated with environment rewards and learned via Soft Actor-Critic, using a pre-trained EEGNet decoder kept fixed for stability. In a Kinova Gen2 pick-and-place task within a cluttered MuJoCo robosuite environment, RLIHF significantly improves performance over sparse rewards and approaches the performance of densely engineered reward baselines, even with moderate decoder accuracy. The work demonstrates robust, scalable human-aligned RL for interactive robotics and highlights the potential of adaptive weighting of implicit neural feedback to balance exploration and guidance.

Abstract

Conventional reinforcement learning (RL) approaches often struggle to learn effective policies under sparse reward conditions, necessitating the manual design of complex, task-specific reward functions. To address this limitation, reinforcement learning from human feedback (RLHF) has emerged as a promising strategy that complements hand-crafted rewards with human-derived evaluation signals. However, most existing RLHF methods depend on explicit feedback mechanisms such as button presses or preference labels, which disrupt the natural interaction process and impose a substantial cognitive load on the user. We propose a novel reinforcement learning from implicit human feedback (RLIHF) framework that utilizes non-invasive electroencephalography (EEG) signals, specifically error-related potentials (ErrPs), to provide continuous, implicit feedback without requiring explicit user intervention. The proposed method adopts a pre-trained decoder to transform raw EEG signals into probabilistic reward components, enabling effective policy learning even in the presence of sparse external rewards. We evaluate our approach in a simulation environment built on the MuJoCo physics engine, using a Kinova Gen2 robotic arm to perform a complex pick-and-place task that requires avoiding obstacles while manipulating target objects. The results show that agents trained with decoded EEG feedback achieve performance comparable to those trained with dense, manually designed rewards. These findings validate the potential of using implicit neural feedback for scalable and human-aligned reinforcement learning in interactive robotics.

Figures (6)

• Figure 1: Conceptual illustration of the proposed RLIHF framework in a real-world pick-and-place scenario. A human user wearing an EEG cap observes a robotic arm navigating a cluttered tabletop environment. The “Original” trajectory (blue) minimizes path length but approaches obstacles too closely, violating implicit spatial preferences. In contrast, the “Modified” trajectory (orange), guided by EEG-based human feedback, maintains safer clearances. EEG signals are decoded to estimate the probability of perceived error, which is transformed into a continuous reward signal used to adapt the behavior of the robot.
• Figure 2: Overview of the proposed RLIHF framework. Implicit human feedback is decoded into scalar rewards and integrated with environment rewards. The resulting rewards guide policy updates via Soft Actor-Critic, with transitions stored in a replay buffer for sample-efficient learning.
• Figure 3: Customized pick-and-place simulation environment used for training and evaluation in the RLIHF framework. A Kinova Gen2 robotic arm operates in a cluttered workspace containing four obstacles (lemon, cereal box, green bottle, and bread). The task requires the robot to grasp the designated target object (red can) and place it at the goal location, marked by a gray cylinder on the right, while avoiding collisions. The environment was built by modifying the Lift task in the robosuite framework and executed within the MuJoCo physics engine.
• Figure 4: Evaluation performance curves for the RL baselines (sparse and dense), and our proposed RLIHF method across 12 human subjects. Each plot shows the mean episodic return during evaluation, averaged across five independent runs. The RLIHF agent consistently outperforms the sparse reward baseline and often approaches the ideal performance achieved with dense rewards, with some inter-subject variability.
• Figure 5: ErrP decoding performance across 12 subjects. Yellow bars indicate accuracy achieved during pretraining, while orange bars represent online performance during real-time feedback integration. While higher decoding accuracy generally correlates with improved RLIHF return (see Fig. \ref{['fig:fig4']}), we observe that performance above chance level is often sufficient to achieve comparable outcomes to dense reward learning.