Does Difficulty even Matter? Investigating Difficulty Adjustment and Practice Behavior in an Open-Ended Learning Task

TL;DR

This paper addresses whether difficulty adjustment affects learning in open-ended tasks, focusing on a graph-theory MIS learning task to compare self-determined difficulty, Dynamic Difficulty Adjustment (DDA), and a predefined progression. It adopts a five-stage online study with 86 participants and analyzes practice behavior from clickstream data through clustering and association rule mining, alongside standard learning and affect measures. A key contribution is the identification of five practice-behavior types and their distinct links to learning outcomes, enabling a nuanced view beyond mere practice performance. The findings show no significant differences between difficulty adjustment methods on normalized learning gain, flow, or perceived difficulty, but reveal that certain practice-behavior types (e.g., thoughtful searchers) achieve higher learning gains, suggesting that incorporating behavior-type analytics could better inform adaptive interventions in open-ended learning environments. Normalized learning gain is defined as $NLG = \begin{cases} \frac{post-pre}{3-pre}, & post>pre \\ \frac{post-pre}{pre}, & post\le pre \end{cases}$, illustrating how improvement is measured relative to potential gains.

Abstract

Difficulty adjustment in practice exercises has been shown to be beneficial for learning. However, previous research has mostly investigated close-ended tasks, which do not offer the students multiple ways to reach a valid solution. Contrary to this, in order to learn in an open-ended learning task, students need to effectively explore the solution space as there are multiple ways to reach a solution. For this reason, the effects of difficulty adjustment could be different for open-ended tasks. To investigate this, as our first contribution, we compare different methods of difficulty adjustment in a user study conducted with 86 participants. Furthermore, as the practice behavior of the students is expected to influence how well the students learn, we additionally look at their practice behavior as a post-hoc analysis. Therefore, as a second contribution, we identify different types of practice behavior and how they link to students' learning outcomes and subjective evaluation measures as well as explore the influence the difficulty adjustment methods have on the practice behaviors. Our results suggest the usefulness of taking into account the practice behavior in addition to only using the practice performance to inform adaptive intervention and difficulty adjustment methods.

Figures (5)

• Figure 1: Main interface of the user study. The screen shows the graph, the buttons for submission and reset, the number of vertices to choose, and the number of vertices the participant already chose. The left figure shows the screen upon arriving on the page, with no vertex selected. The right figure shows a valid solution to the graph.
• Figure 2: Stages of the user study.
• Figure 3: The clustering method. The practice behavior is extracted from recorded practice sessions, and then used for clustering. We compare the evaluation measures on the clusters to validate the differences between clusters and mine rules to see the defining characteristics of each cluster.
• Figure 4: Comparison of different learning outcomes between the three conditions. The plots show from left to right, learning gain, flow, and perceived difficulty of each cluster of students.
• Figure 5: Results from clustering by practice behavior. The plots show from left to right, the learning gain, flow, and perceived difficulty of each cluster of students. The learning gain plot also shows significant pairs of differences after the Benjamini-Hochberg correction. The sizes of the clusters are 24, 14, 9, 8, 20, respectively.