Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education

TL;DR

The paper addresses the gap between CS education research and classroom practice by exploring adaptive experiments that continually improve learning experiences. It adopts a Beta-Bernoulli Thompson Sampling ($TS$-$BB$) framework to allocate learning-support conditions in real time, using proximal rewards to drive online updates. The case study in CS1 comparing adaptive and traditional A/B designs demonstrates substantial reallocation of students toward better conditions (e.g., 82.6% to the preferred arm) and discusses benefits and limitations of this approach, including statistical inference with bandit data and potential personalization. The work highlights the practical impact of adaptive experimentation for continuous course improvement and advocates for supportive infrastructures and partnerships to scale these methods in computing education.

Abstract

Randomized A/B comparisons of alternative pedagogical strategies or other course improvements could provide useful empirical evidence for instructor decision-making. However, traditional experiments do not provide a straightforward pathway to rapidly utilize data, increasing the chances that students in an experiment experience the best conditions. Drawing inspiration from the use of machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might aid continuous course improvement. In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. This can be achieved using machine learning algorithms to identify which actions are more beneficial in improving students' learning experiences and outcomes. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students, resulting in better support for students' needs. We illustrate this approach with a case study that provides a side-by-side comparison of traditional and adaptive experiments on adding self-explanation prompts in online homework problems in a CS1 course. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement in educational settings.

Figures (3)

• Figure 1: Schematic implementation of Thompson Sampling Beta Bernoulli algorithm (TS-BB), based on chapelle_empirical_2011.
• Figure 2: Multiple-choice Questions connected to the drop-down in the online homework system.
• Figure 3: Design of the self-explanation prompt drop-down.