Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning

Christopher James MacLellan, Kimberly Stowers, Lisa Brady

TL;DR

This paper addresses the problem that costly A/B experiments impede efficient evaluation of training interventions. It proposes personalized Apprentice Learner cognitive agents tuned with HyperOpt to predict learning outcomes and generate counterfactual predictions for alternative fractions tutoring interventions. The key contributions are showing that personalized agents predict target students better than generic ones, demonstrating counterfactual simulations for high and low performing students across blocked, interleaved, and faded interventions, and aligning predictions with known human findings while offering testable hypotheses for future experiments. The work offers a practical, low cost design tool for instructional designers and tutors to explore intervention space before conducting expensive human trials.

Abstract

Evaluating different training interventions to determine which produce the best learning outcomes is one of the main challenges faced by instructional designers. Typically, these designers use A/B experiments to evaluate each intervention; however, it is costly and time consuming to run such studies. To address this issue, we explore how computational models of learning might support designers in reasoning causally about alternative interventions within a fractions tutor. We present an approach for automatically tuning models to specific individuals and show that personalized models make better predictions of students' behavior than generic ones. Next, we conduct simulations to generate counterfactual predictions of performance and learning for two students (high and low performing) in different versions of the fractions tutor. Our approach makes predictions that align with previous human findings, as well as testable predictions that might be evaluated with future human experiments.

Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning

TL;DR

This paper addresses the problem that costly A/B experiments impede efficient evaluation of training interventions. It proposes personalized Apprentice Learner cognitive agents tuned with HyperOpt to predict learning outcomes and generate counterfactual predictions for alternative fractions tutoring interventions. The key contributions are showing that personalized agents predict target students better than generic ones, demonstrating counterfactual simulations for high and low performing students across blocked, interleaved, and faded interventions, and aligning predictions with known human findings while offering testable hypotheses for future experiments. The work offers a practical, low cost design tool for instructional designers and tutors to explore intervention space before conducting expensive human trials.

Abstract

Evaluating different training interventions to determine which produce the best learning outcomes is one of the main challenges faced by instructional designers. Typically, these designers use A/B experiments to evaluate each intervention; however, it is costly and time consuming to run such studies. To address this issue, we explore how computational models of learning might support designers in reasoning causally about alternative interventions within a fractions tutor. We present an approach for automatically tuning models to specific individuals and show that personalized models make better predictions of students' behavior than generic ones. Next, we conduct simulations to generate counterfactual predictions of performance and learning for two students (high and low performing) in different versions of the fractions tutor. Our approach makes predictions that align with previous human findings, as well as testable predictions that might be evaluated with future human experiments.
Paper Structure (14 sections, 7 figures)

This paper contains 14 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Fraction Arithmetic Tutor
  4. Apprentice Learner Architecture
  5. Tuning a Model to Account for Individual Student Differences
  6. Our Model Tuning Approach
  7. Evaluation of Model Tuning
  8. Generating Counterfactual Prediction with Personalized Models
  9. Evaluating the Use of Personalized Models for Counterfactual Prediction
  10. High-Performing Student Predictions
  11. Low-Performing Student Predictions
  12. Discussion
  13. Related Research
  14. Conclusions and Future Work

Figures (7)

  • Figure 1: Three types of problems presented to students by the fractions tutoring system.
  • Figure 2: A conceptual depiction of the memories (blue boxes), performance components (yellow diamonds), and learning components (green circles) for our Apprentice Learner agents.
  • Figure 3: The iterative process used to tune an Apprentice agents to better model a specific student.
  • Figure 4: (a) Prediction Error for Apprentice Learner Models that are trained and tested on the same data. (b) Prediction Error for Apprentice Learner Models that are trained on earlier data and tested on later unseen data (within individuals). In both cases, the error bars represent the 95% confidence intervals.
  • Figure 5: Conceptual depiction of three possible orderings for the fraction arithmetic tutor. Shadings of cells denote the three types of problems: fraction addition with same denominators, fraction arithmetic with different denominators, and fraction multiplication.
  • ...and 2 more figures