Detecting Student Intent for Chat-Based Intelligent Tutoring Systems

TL;DR

This paper tackles detecting student intent within a chat-based intelligent tutoring system by focusing on whether a learner wants to continue the current lesson or change topics. It compares four modeling approaches—random forest with TF-IDF, a fine-tuned Llama 2 with LoRA, OpenAI LLMs with prompts, and a function-calling variant—on a dataset of 2,101 math-history conversations from the Rori ITS. Results show modest overall performance, with a traditional classifier at F1 ≈ 0.62 and a fine-tuned Llama 2 reaching F1 ≈ 0.66, while larger OpenAI models achieve higher accuracy but incur higher latency; function calling offers similar performance to prompting. The study highlights the trade-off between accuracy and response time in live ITS and discusses practical deployment considerations, such as model confidence calibration and the need for broader intent design and pedagogy-aligned responses in chat interfaces.

Abstract

Chat interfaces for intelligent tutoring systems (ITSs) enable interactivity and flexibility. However, when students interact with chat interfaces, they expect dialogue-driven navigation from the system and can express frustration and disinterest if this is not provided. Intent detection systems help students navigate within an ITS, but detecting students' intent during open-ended dialogue is challenging. We designed an intent detection system in a chatbot ITS, classifying a student's intent between continuing the current lesson or switching to a new lesson. We explore the utility of four machine learning approaches for this task - including both conventional classification approaches and fine-tuned large language models - finding that using an intent classifier introduces trade-offs around implementation cost, accuracy, and prediction time. We argue that implementing intent detection in chat interfaces can reduce frustration and support student learning.