Simulated Human Learning in a Dynamic, Partially-Observed, Time-Series Environment

TL;DR

This work introduces SimEdu, a configurable dynamic time-series environment for evaluating intelligent tutoring systems under partial observability. It combines population-informed knowledge tracing via a Dirichlet-HMM with reinforcement learning and heuristic policies to study probing strategies, course structures, and information gain. Key findings show that probing enhances policy performance and that more information-rich course structures improve outcomes, while RL does not dramatically outperform well-designed heuristics in these settings. The framework provides a flexible platform for exploring probing costs, population shifts, and intervention tradeoffs with potential applicability to other time-series domains beyond education.

Abstract

While intelligent tutoring systems (ITSs) can use information from past students to personalize instruction, each new student is unique. Moreover, the education problem is inherently difficult because the learning process is only partially observable. We therefore develop a dynamic, time-series environment to simulate a classroom setting, with student-teacher interventions - including tutoring sessions, lectures, and exams. In particular, we design the simulated environment to allow for varying levels of probing interventions that can gather more information. Then, we develop reinforcement learning ITSs that combine learning the individual state of students while pulling from population information through the use of probing interventions. These interventions can reduce the difficulty of student estimation, but also introduce a cost-benefit decision to find a balance between probing enough to get accurate estimates and probing so often that it becomes disruptive to the student. We compare the efficacy of standard RL algorithms with several greedy rules-based heuristic approaches to find that they provide different solutions, but with similar results. We also highlight the difficulty of the problem with increasing levels of hidden information, and the boost that we get if we allow for probing interventions. We show the flexibility of both heuristic and RL policies with regards to changing student population distributions, finding that both are flexible, but RL policies struggle to help harder classes. Finally, we test different course structures with non-probing policies and we find that our policies are able to boost the performance of quiz and midterm structures more than we can in a finals-only structure, highlighting the benefit of having additional information.

Figures (6)

• Figure 1: A figure showing the time-dynamics of the interactions between the three components of SimEdu: The population model, the RL agent, and the Simulated Environment.
• Figure 2: Concept DAGs for multi-concept courses (left: one-concept course with prerequisite, right: four-concept course with two prerequisites). Concepts taught in a course are denoted with the precursor C and are denoted by letters (e.g. CA), while prerequisites are denoted with precursor PR and selected via numbers (e.g. PR1).
• Figure 3: Bayesian knowledge tracing via a Dirichlet parameterized sampling technique. The population uses state information (which can possibly come from observations, such as the RL's choice of action) to sample $P_T \sim Dirichlet(\phi_T^{(o_t)})$. After sampling, we proceed with knowledge tracing with standard HMM updates. We can also sample a $P_E$ in a similar way, but in our case we use $P_E$ as fixed priors.
• Figure 4: The time-dynamic DAG structure of the four-concept course with the concept DAG referenced in \ref{['fig:concept-graph']}. The structure experiments are defined based on which subset of evaluations ($Q_1$, $M$, $Q_2$, and $F$) are present.
• Figure 5: Example DQN Training Trajectory for DQN without probing capabilities on an unobserved course. Error bars represent the one standard deviation away from the mean across 1000 simulated students.