RelianceScope: An Analytical Framework for Examining Students' Reliance on Generative AI Chatbots in Problem Solving

TL;DR

RelianceScope is introduced, an analytical framework that characterizes students'reliance on chatbots during problem-solving and enables fine-grained analysis of reliance in open-ended student-AI interactions, and the implications of RelianceScope and the design guidelines for AI-supported educational systems are discussed.

Abstract

Generative AI chatbots enable personalized problem-solving, but effective learning requires students to self-regulate both how they seek help and how they use AI-generated responses. Considering engagement modes across these two actions reveals nuanced reliance patterns: for example, a student may actively engage in help-seeking by clearly specifying areas of need, yet engage passively in response-use by copying AI outputs, or vice versa. However, existing research lacks systematic tools for jointly capturing engagement across help-seeking and response-use, limiting the analysis of such reliance behaviors. We introduce RelianceScope, an analytical framework that characterizes students' reliance on chatbots during problem-solving. RelianceScope (1) operationalizes reliance into nine patterns based on combinations of engagement modes in help-seeking and response-use, and (2) situates these patterns within a knowledge-context lens that accounts for students' prior knowledge and the instructional significance of knowledge components. Rather than prescribing optimal AI use, the framework enables fine-grained analysis of reliance in open-ended student-AI interactions. As an illustrative application, we applied RelianceScope to analyze chat and code-edit logs from 79 college students in a web programming course. Results show that active help-seeking is associated with active response-use, whereas reliance patterns remain similar across knowledge mastery levels. Students often struggled to articulate their knowledge gaps and to adapt AI responses. Using our annotated dataset as a benchmark, we further demonstrate that large language models can reliably detect reliance during help-seeking and response-use. We conclude by discussing the implications of RelianceScope and the design guidelines for AI-supported educational systems.

Figures (5)

• Figure 1: An illustration of how to classify reliance patterns in a programming learning activity. Purple boxes indicate interaction segments that group chat messages and code edit logs. Green annotations show engagement modes at the individual message level, which are aggregated into a single reliance pattern for each interaction segment.
• Figure 2: The interface used in the data collection study. The activity's learning objective and step-by-step guidance are always displayed on the left. Students could use a code editor to work and check results and console errors in real-time in the middle section. Students could also seek help from a chatbot on the right. A timer is located in the top-right corner.
• Figure 3: Flow of engagement modes from help-seeking to response use, with band widths representing proportions of interaction segments.
• Figure 4: Distribution of reliance patterns across three knowledge contexts. Engagement modes on the x-axis are abbreviated as P (Passive), A (Active), and C (Constructive). The proportion of the P_P, P_A, and P_C patterns differ significantly between Focal and Supporting.
• Figure 5: Transitions between reliance patterns. Engagement modes are abbreviated as P (Passive), A (Active), and C (Constructive).