Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fitting Reinforcement Learning Model to Behavioral Data under Bandits

Hao Zhu, Jasper Hoffmann, Baohe Zhang, Joschka Boedecker

TL;DR

This work tackles fitting reinforcement learning models to behavioral data collected in multi-armed bandit tasks by formulating a general optimization problem and analyzing its convexity. It introduces a convex surrogate that relaxes a non-affine transformation to enable tractable, globally solvable optimization, while still providing useful lower bounds on the original problem. The approach is implemented in an open-source Python package (rlfit) and validated on simulated 2-arm and 10-arm bandit environments across basic, per-arm, and subreward extensions, showing comparable fitting accuracy to benchmarks with substantially faster computation. The results offer practical guidance for researchers to characterize decision-making behavior and provide a principled basis to evaluate heuristic fits via provable bounds.

Abstract

We consider the problem of fitting a reinforcement learning (RL) model to some given behavioral data under a multi-armed bandit environment. These models have received much attention in recent years for characterizing human and animal decision making behavior. We provide a generic mathematical optimization problem formulation for the fitting problem of a wide range of RL models that appear frequently in scientific research applications, followed by a detailed theoretical analysis of its convexity properties. Based on the theoretical results, we introduce a novel solution method for the fitting problem of RL models based on convex relaxation and optimization. Our method is then evaluated in several simulated bandit environments to compare with some benchmark methods that appear in the literature. Numerical results indicate that our method achieves comparable performance to the state-of-the-art, while significantly reducing computation time. We also provide an open-source Python package for our proposed method to empower researchers to apply it in the analysis of their datasets directly, without prior knowledge of convex optimization.

Fitting Reinforcement Learning Model to Behavioral Data under Bandits

TL;DR

This work tackles fitting reinforcement learning models to behavioral data collected in multi-armed bandit tasks by formulating a general optimization problem and analyzing its convexity. It introduces a convex surrogate that relaxes a non-affine transformation to enable tractable, globally solvable optimization, while still providing useful lower bounds on the original problem. The approach is implemented in an open-source Python package (rlfit) and validated on simulated 2-arm and 10-arm bandit environments across basic, per-arm, and subreward extensions, showing comparable fitting accuracy to benchmarks with substantially faster computation. The results offer practical guidance for researchers to characterize decision-making behavior and provide a principled basis to evaluate heuristic fits via provable bounds.

Abstract

We consider the problem of fitting a reinforcement learning (RL) model to some given behavioral data under a multi-armed bandit environment. These models have received much attention in recent years for characterizing human and animal decision making behavior. We provide a generic mathematical optimization problem formulation for the fitting problem of a wide range of RL models that appear frequently in scientific research applications, followed by a detailed theoretical analysis of its convexity properties. Based on the theoretical results, we introduce a novel solution method for the fitting problem of RL models based on convex relaxation and optimization. Our method is then evaluated in several simulated bandit environments to compare with some benchmark methods that appear in the literature. Numerical results indicate that our method achieves comparable performance to the state-of-the-art, while significantly reducing computation time. We also provide an open-source Python package for our proposed method to empower researchers to apply it in the analysis of their datasets directly, without prior knowledge of convex optimization.

Paper Structure

This paper contains 34 sections, 39 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Multi-armed bandit behavioral tasks.
  3. RL model for decision making under bandits.
  4. RL model fitting problem.
  5. This paper.
  6. The fitting problem of RL models
  7. Basic forgetting Q-learning model
  8. Extensions
  9. Extension on reward signals.
  10. Multiple learning rates and reward sensitivity.
  11. Subreward signals and subvalue functions.
  12. RL model fitting problems in general form
  13. Convexity properties
  14. Solution method
  15. The convex surrogate
  16. ...and 19 more sections

Figures (2)

  • Figure 1: Performance of different solution methods under the 2AB environment.
  • Figure 2: Performance of different solution methods under the 10AB environment.