Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

TL;DR

The paper addresses the fragmented landscape of human feedback in reinforcement learning and proposes a unified framework that bridges human–computer interaction with RL innovation. It introduces a nine-dimension taxonomy spanning human-, interface-, and model-centered aspects, along with seven quality metrics to evaluate feedback. A formalization of feedback processing is provided, including an explicit translation pipeline and target/measurement constructs that map human input to a learnable reward signal. Based on these foundations, the authors outline design requirements for RL systems that can harvest diverse, expressive feedback and discuss opportunities for interdisciplinary collaboration to unlock data-driven co-adaptive learning. The work aims to broaden RLHF beyond demonstrations and simple comparisons, enabling richer, context-aware human guidance across training and deployment stages with potential impact on alignment and efficiency of intelligent agents.

Abstract

Reinforcement Learning from Human feedback (RLHF) has become a powerful tool to fine-tune or train agentic machine learning models. Similar to how humans interact in social contexts, we can use many types of feedback to communicate our preferences, intentions, and knowledge to an RL agent. However, applications of human feedback in RL are often limited in scope and disregard human factors. In this work, we bridge the gap between machine learning and human-computer interaction efforts by developing a shared understanding of human feedback in interactive learning scenarios. We first introduce a taxonomy of feedback types for reward-based learning from human feedback based on nine key dimensions. Our taxonomy allows for unifying human-centered, interface-centered, and model-centered aspects. In addition, we identify seven quality metrics of human feedback influencing both the human ability to express feedback and the agent's ability to learn from the feedback. Based on the feedback taxonomy and quality criteria, we derive requirements and design choices for systems learning from human feedback. We relate these requirements and design choices to existing work in interactive machine learning. In the process, we identify gaps in existing work and future research opportunities. We call for interdisciplinary collaboration to harness the full potential of reinforcement learning with data-driven co-adaptive modeling and varied interaction mechanics.

Figures (6)

• Figure 1: Human-AI Interaction via Interactive Communication Interfaces: Modal Outputs are communicated to the humans via the interface and presented as comprehensible explanations that should preserve high fidelity. Human feedback is communicated via interactions that must be mapped to a reward signal suitable for RL training.
• Figure 2: The methodology used to create the conceptual framework: During the initialization, we extract a series of initial dimensions and qualities that exhaustively classify a set of highly relevant surveyed papers. In a second refinement phase, we iteratively refine the conceptual framework based on additional papers, external input, and internal discussions. Importantly, the framework is also implemented as a software system and validated to handle multiple use cases. Finally, we validate the refined conceptual framework via the implementation and in expert interviews.
• Figure 3: The feedback process formalized: Humans generate feedback for agents that they observe acting in an environment. A feedback instance is based on a target and measured values. It is translated into a form that can be used to update a reward model. The policy of the agent, in turn, is optimized with respect to the updated reward model.
• Figure 4: The feedback state consists of three different sub-states: (1) The HumanColorhuman feedback state, (2) the state of the CombinedColorinterface, and (3) the ModelColormodel state.
• Figure 5: In the context of Reinforcement Learning, we present a conceptual framework of human feedback based on nine dimensions: We identify HumanColorhuman-centered, CombinedColorinterface-centered and ModelColormodel-centered dimension.