Gaining Insights into Group-Level Course Difficulty via Differential Course Functioning

Abstract

Curriculum Analytics (CA) studies curriculum structure and student data to ensure the quality of educational programs. One desirable property of courses within curricula is that they are not unexpectedly more difficult for students of different backgrounds. While prior work points to likely variations in course difficulty across student groups, robust methodologies for capturing such variations are scarce, and existing approaches do not adequately decouple course-specific difficulty from students' general performance levels. The present study introduces Differential Course Functioning (DCF) as an Item Response Theory (IRT)-based CA methodology. DCF controls for student performance levels and examines whether significant differences exist in how distinct student groups succeed in a given course. Leveraging data from over 20,000 students at a large public university, we demonstrate DCF's ability to detect inequities in undergraduate course difficulty across student groups described by grade achievement. We compare major pairs with high co-enrollment and transfer students to their non-transfer peers. For the former, our findings suggest a link between DCF effect sizes and the alignment of course content to student home department motivating interventions targeted towards improving course preparedness. For the latter, results suggest minor variations in course-specific difficulty between transfer and non-transfer students. While this is desirable, it also suggests that interventions targeted toward mitigating grade achievement gaps in transfer students should encompass comprehensive support beyond enhancing preparedness for individual courses. By providing more nuanced and equitable assessments of academic performance and difficulties experienced by diverse student populations, DCF could support policymakers, course articulation officers, and student advisors.

Figures (6)

• Figure 1: DCF model. The green sigmoid function indicates the response curve of a Rasch model fitted for all students. The red and blue curves indicate group-specific course responses (red $\sim -1$, blue $\sim 1$) that exhibit asymmetric offsets from the Rasch model. Parameter $\beta_{c,0}$ is the intercept of the logistic regression model and captures the distance between the Rasch IRF and the position on the x-axis from which both group IRFs are equidistant. This is the reference point for the group-specific DCF measure $\beta_{c,1}$ (difficulty disparity).
• Figure 2: Possible relationships between AR differences ($\text{AR}_{G_1}$, $\text{AR}_{G_2}$) and DCF for groups ($G_1$, $G_2$) with identical and deviating student traits values ($\theta_{G_1}$, $\theta_{G_2}$). DCF offers a more nuanced measure of academic performance to gain insights into difficulties experienced by diverse student populations.
• Figure 3: Examples of courses illustrating the relationships between AR and DCF outlined in Figure \ref{['fig:dcf_pr_cases']}. By decoupling course difficulty and student performance levels, the DCF methodology can yield more robust insights into disparities in course difficulties compared to analysis of AR differences.
• Figure 4: DCF effects for Major/Minor Pairings. [Top] Economics and Computer Science, [Bottom] Applied Math and Computer Science. Course names include the department and a number abbreviation (e.g., "Statistics140"). Lower division courses have a number less than or equal to 100. Otherwise, they are upper-division courses. Green bars indicate a significant effect. A negative effect means that the course is easier for the first major in the tuple, and positive effects are the opposite. The effect size is the percentage difference between the groups at the average student trait value across both groups. If there is an effect of $0.2$, the student in the preferred group with an average trait value of both groups is $20\%$ more likely to receive a response of $1$.
• Figure 5: Comparison of [Top] DCF effects, similar to Figure \ref{['fig:dif_major_minor']}, and [Bottom] course AR group differences for Transfer Status of the Economics major. Again, the DCF effect sizes ([Top]) are the differences in passing probability between the groups at the average student trait value across both groups. The results indicate that DCF is more robust than ARs due to mitigation of student performance. Differences in effect significance and effect size between DCF and ARs in the same courses indicate that DCF classifies far fewer effects as significant, and effect sizes can differ greatly between measures.