Model selection for behavioral learning data and applications to contextual bandits

TL;DR

This work tackles model selection for behavioral learning data characterized by non-stationarity and dependency. It introduces a hold-out approach with an oracle inequality and an AIC-type penalized likelihood criterion, both adapted to dependent data, and provides their theoretical guarantees. The methods are instantiated within partition-based contextual bandits to compare models of how individuals learn from contextual feedback, with theoretical bounds and practical demonstrations on synthetic and real categorization data. The results offer principled tools for cognitive scientists to infer learning mechanisms from single learning trajectories, while highlighting trade-offs and tuning considerations for real-world applications.

Abstract

Learning for animals or humans is the process that leads to behaviors better adapted to the environment. This process highly depends on the individual that learns and is usually observed only through the individual's actions. This article presents ways to use this individual behavioral data to find the model that best explains how the individual learns. We propose two model selection methods: a general hold-out procedure and an AIC-type criterion, both adapted to non-stationary dependent data. We provide theoretical error bounds for these methods that are close to those of the standard i.i.d. case. To compare these approaches, we apply them to contextual bandit models and illustrate their use on both synthetic and experimental learning data in a human categorization task.

Figures (5)

• Figure 1: Experiment presentation: classic $5$-$4$ category structure, widely used in cognition medin1978context. In \ref{['fig:4Dspace']}, the 9 objects to classify represented in a 4D space with respect to their attributes: Color, Size, Filling Pattern, and Shape. In \ref{['fig:categoryattribution']}, by position in the 4D space, the category attribution (A or B).
• Figure 2: Errors of the procedures as a function of the tuning parameters. In \ref{['fig:boxplot_estimationerror']}, average of the $|\hat{\theta}_C-\theta_C|/\theta_C$ over all cells $C$ in model OneForAll and OnePerItem for the data generated respectively by the same models, where $\hat{\theta}_C$ is the MLE with likelihood truncated at $N$ (in abscissa). In \ref{['fig:choiceofN']} and \ref{['fig:choiceofc']}, percentage of mismatch between $\hat{m}$ and the simulated model over 100 simulations. The colors for each model are the ones given in Figure \ref{['fig:boxplot_holdout_pmle']} whereas the average error on the models in the dash line. In \ref{['fig:choiceofN']}, for the hold-out estimator as a function of $N/n$. In \ref{['fig:choiceofc']}, for the penalized MLE with $\mathop{\mathrm{\text{pen}}}\nolimits(m)=c\log(n)^2D_m/n$, as a function of $c$.
• Figure 3: Distribution of the model choices. In a, hold-out with $N=250$ over 100 simulations. In b, penalized MLE with $c=0.012$ over 100 simulations. In c, hold-out on the data recorded in mezzadri2022investigating--176 participants. In d, penalized MLE on the same experimental data.
• Figure 4: Design of models ByPattern, ByPatternExc and ByShapeExc. Exceptions are determined by the categorization rule in Figure 1b of the article.
• Figure 5: Errors of the penalized log-likelihood criterion as a function of the tuning parameter $c$, where $c$ is such that $\mathop{\mathrm{\text{pen}}}\nolimits(m)=c\log(n)^2D_m/n$ is a function of $c$. Both figures show the percentage of mismatches between $\hat{m}$ and the simulated model over 100 simulations. The same simulations were used for both figures. The evolution of errors as a function of $c$ is logical in relation to the value of $D_m$.

Theorems & Definitions (18)

• Theorem 1
• Proposition 2
• Theorem 3
• Proposition 4
• Proposition 5
• Proposition 6
• proof : Proof of Theorem \ref{['holdouttheorem']}
• Lemma 7
• proof
• Lemma 8