Contextual bandits with entropy-based human feedback

TL;DR

The paper addresses the challenge of leveraging explicit human feedback in contextual bandits under model uncertainty and variable feedback quality. It introduces an entropy-based solicitation mechanism that queries an expert only when the policy's uncertainty, quantified by $H(\pi_t)$, exceeds a threshold $\lambda$, and supports two feedback modes: Action Recommendation (AR) and Reward Manipulation (RM). A theoretical regret bound is derived, showing a trade-off between exploration and feedback frequency, and extensive experiments on multiple datasets demonstrate that entropy-based feedback reduces cumulative regret and maintains robustness even with imperfect expert quality. The results provide practical guidance for scalable human-in-the-loop CB systems and offer a general, model-agnostic framework for integrating human guidance into sequential decision-making tasks, with code publicly available.

Abstract

In recent years, preference-based human feedback mechanisms have become essential for enhancing model performance across diverse applications, including conversational AI systems such as ChatGPT. However, existing approaches often neglect critical aspects, such as model uncertainty and the variability in feedback quality. To address these challenges, we introduce an entropy-based human feedback framework for contextual bandits, which dynamically balances exploration and exploitation by soliciting expert feedback only when model entropy exceeds a predefined threshold. Our method is model-agnostic and can be seamlessly integrated with any contextual bandit agent employing stochastic policies. Through comprehensive experiments, we show that our approach achieves significant performance improvements while requiring minimal human feedback, even under conditions of suboptimal feedback quality. This work not only presents a novel strategy for feedback solicitation but also highlights the robustness and efficacy of incorporating human guidance into machine learning systems. Our code is publicly available: https://github.com/BorealisAI/CBHF

Figures (9)

• Figure 1: Overview of the architecture. Our framework builds upon a standard contextual bandit setup. The right side of the figure illustrates the human feedback incorporation mechanism, which can be integrated through either reward manipulation (directly modifying the bandit's reward signal) or action recommendation, which constrains the set of available actions. Rather than triggering human feedback at fixed intervals, we propose an adaptive approach: querying feedback only when the action policy (Eq. \ref{['eq:policy_entropy']}) exceeds a predefined uncertainty threshold. This strategy ensures feedback is solicited when it is most valuable, improving efficiency and decision-making.
• Figure 2: Comparison of expert feedback for different learners based on different expert qualities. The results show that mean cumulative reward for different datasets and algorithms vary in a different manner for the two feedback schemes considered. Higher levels of expert does not necessary results in better performance.
• Figure 3: Performance comparison of baselines and the proposed schemes. The figures show that using entropy based feedback leads to lower mean cumulative regret. The solid line represents the mean cumulative regret and the shaded region represents the $\pm$ 1 standard deviation across the mean.
• Figure 4: Comparison of expert feedback for different learners based on different expert qualities. The results show that mean cumulative regret for different datasets and algorithms vary in a different manner for the two feedback schemes considered. Higher levels of expert does not necessary results in better performance.
• Figure 5: Comparison of model performance for different values of entropy and expert accuracies for feedback: Action Recommendation and Reward Manipulation. The size and color of each bubble in the bubble plots represent the magnitude of the mean cumulative reward.