Visualizing Intelligent Tutor Interactions for Responsive Pedagogy

TL;DR

The paper addresses the challenge of integrating intelligent tutoring systems into curriculum to support responsive pedagogy. It introduces VisTA, a visual analytics tool that visualizes fine-grained apprenticeship tutor logs using four coordinated views, grounded in a design study with five teachers deploying Apprentice Tutors. The qualitative evaluation shows that VisTA helps teachers interpret student problem-solving provenance, identify knowledge component steps needing instruction, and tailor follow-up actions, with potential to support self-directed learning. The work offers a practical path to embedding ITS data into classroom planning and suggests extensions to sequence detection and student-directed learning in future deployments.

Abstract

Intelligent tutoring systems leverage AI models of expert learning and student knowledge to deliver personalized tutoring to students. While these intelligent tutors have demonstrated improved student learning outcomes, it is still unclear how teachers might integrate them into curriculum and course planning to support responsive pedagogy. In this paper, we conducted a design study with five teachers who have deployed Apprentice Tutors, an intelligent tutoring platform, in their classes. We characterized their challenges around analyzing student interaction data from intelligent tutoring systems and built VisTA (Visualizations for Tutor Analytics), a visual analytics system that shows detailed provenance data across multiple coordinated views. We evaluated VisTA with the same five teachers, and found that the visualizations helped them better interpret intelligent tutor data, gain insights into student problem-solving provenance, and decide on necessary follow-up actions - such as providing students with further support or reviewing skills in the classroom. Finally, we discuss potential extensions of VisTA into sequence query and detection, as well as the potential for the visualizations to be useful for encouraging self-directed learning in students.

Figures (5)

• Figure 1: Top: An example of two rows of data collected by Apprentice Tutors. Bottom: How data columns map to different parts of the system. Each row records a single interaction by a student. From left to right, each interaction has 11 attributes: ID, userID (anonymized student ID), tutor, interface (problem type), start_state (problem), selection (input box/step being attempted), action (interaction type), input (student input), correctness (correctness of input), kc_labels (knowledge component/skill associated with this step), and time (recorded time of student interaction).
• Figure 2: The Overview visualization showing proportion of correct (green), incorrect (red), and skipped (gray) questions attempted by students for each problem type. Users can click on the respective labels in the legend to select and deselect the groups they would like to include in the histogram. The control panel on the left can be used to select a particular problem type or student to visualize.
• Figure 3: The Student view showing all problems of a particular problem type attempted by a single student. We can see that the student spent the most time and made the most mistakes on the first two problems, but solved subsequent problems quickly and more accurately. When new problem types with new knowledge components (solving polynomials with first coefficient $>$1) were introduced, the student started to make more mistakes and left the problems incomplete.
• Figure 4: Left: The Problem Type view uses a step line chart to depict all student attempts of a particular problem type. In this example, we see that while most attempts were completed in under 5 minutes, there was one student attempt that took more than 15 minutes and was left incomplete. Right: Clicking on a path brings up the Details view, which shows the breakdown of each step. Here, the student appears to have made multiple mistakes on the "new_first_expression" and "new_second_expression" steps before leaving the problem incomplete.
• Figure 5: Student view for a student whose interaction patterns show that they frequently asked for hints while working through the problems.