Interpretable Knowledge Tracing via Response Influence-based Counterfactual Reasoning

TL;DR

The paper tackles the interpretability gap in deep learning knowledge tracing by introducing RCKT, a response influence-based counterfactual framework. RCKT explains predictions through quantified influences of individual past responses on future questions, using a monotonicity-based counterfactual construction and a bi-directional adaptive encoder-MLP to estimate probabilities. It also introduces an efficient approximation to reduce inference time and a joint training regime to enhance robustness and interpretability. Experiments across four KT datasets show that RCKT achieves competitive or superior accuracy while providing credible, example-driven explanations of response influences that can aid educators in understanding student learning dynamics.

Abstract

Knowledge tracing (KT) plays a crucial role in computer-aided education and intelligent tutoring systems, aiming to assess students' knowledge proficiency by predicting their future performance on new questions based on their past response records. While existing deep learning knowledge tracing (DLKT) methods have significantly improved prediction accuracy and achieved state-of-the-art results, they often suffer from a lack of interpretability. To address this limitation, current approaches have explored incorporating psychological influences to achieve more explainable predictions, but they tend to overlook the potential influences of historical responses. In fact, understanding how models make predictions based on response influences can enhance the transparency and trustworthiness of the knowledge tracing process, presenting an opportunity for a new paradigm of interpretable KT. However, measuring unobservable response influences is challenging. In this paper, we resort to counterfactual reasoning that intervenes in each response to answer \textit{what if a student had answered a question incorrectly that he/she actually answered correctly, and vice versa}. Based on this, we propose RCKT, a novel response influence-based counterfactual knowledge tracing framework. RCKT generates response influences by comparing prediction outcomes from factual sequences and constructed counterfactual sequences after interventions. Additionally, we introduce maximization and inference techniques to leverage accumulated influences from different past responses, further improving the model's performance and credibility. Extensive experimental results demonstrate that our RCKT method outperforms state-of-the-art knowledge tracing methods on four datasets against six baselines, and provides credible interpretations of response influences.

Figures (6)

• Figure 1: A toy example comparing inference between conventional DLKT models (upper dashed box) and models with response influences (lower dashed box). The arrow lines with green and red values represent the influence of past correct and incorrect responses, respectively, on the target question. The question description is displayed in the middle. This illustrative example is derived from the Eedi dataset, and more details about the dataset can be referred to in Sec. \ref{['subsec:experimental-setup']}.
• Figure 2: The framework of RCKT. $\textbf{F+}$ denotes the factual response sequence when we assume the target question is answered correctly. $\textbf{CF-}$ denotes the counterfactual response sequence when we intervene in the response to the target question in $\textbf{F+}$ to be incorrect. $\textbf{F-}$ and $\textbf{CF+}$ denote the sequences vice versa. The meanings of other symbols are exhibited on the top and right.
• Figure 3: The process of constructing counterfactual sequences by the monotonicity assumption. We use the sample example in Fig. \ref{['fig:intro']}. Thresholds determine whether the knowledge proficiency is enough to make a correct answer.
• Figure 4: The performance of RCKT-DKT and RCKT-AKT with different loss balancers on ASSIST09 and ASSIST12.
• Figure 5: Interpretable knowledge proficiency tracking of an ASSIST12 student by RCKT. Each color corresponds to a different knowledge concept as depicted on the top. The circles represent the student’s responses, with solid ones for correct responses and hollow ones for incorrect responses. The colored octagons on the bottom represent the response influences on capturing the corresponding concepts. To make the comparison more obvious, we negate the influences of incorrect responses. The colored squares on the top indicate the dynamic proficiency of the corresponding concepts after responding to each question, whose values are scaled into $(0,1)$.