Exploring The Interaction-Outcome Paradox: Seemingly Richer and More Self-Aware Interactions with LLMs May Not Yet Lead to Better Learning

TL;DR

The paper investigates whether LLM-based learning interfaces outperform search-based interfaces in higher-education tasks. Using a within-subjects design with $N=20$ participants to compare ChatGPT and Google across two topics, it analyzes input, output, and outcome through qualitative coding, automated text metrics, and expert rubrics. The key finding is an Interaction-Outcome Paradox: LLMs elicit richer, more metacognitively articulated input but do not yield broadly superior learning outcomes. The study discusses cognitive shifts in how learners allocate effort, with implications for designing educational technologies that scaffold productive cognitive work while addressing AI literacy and assessment fairness, including differing plagiarism signals between tool types.

Abstract

While Large Language Models (LLMs) have transformed the user interface for learning, moving from keyword search to natural language dialogue, their impact on educational outcomes remains unclear. We present a controlled study (N=20) that directly compares the learning interaction and outcomes between LLM and search-based interfaces. We found that although LLMs elicit richer and nuanced interactions from a learner, they do not produce broadly better learning outcomes. In this paper, we explore this the ``Interaction-Outcome Paradox.'' To explain this, we discuss the concept of a cognitive shift: the locus of student effort moves from finding and synthesizing disparate sources (search) to a more self-aware identification and articulation of their knowledge gaps and strategies to bridge those gaps (LLMs). This insight provides a new lens for evaluating educational technologies, suggesting that the future of learning tools lies not in simply enriching interaction, but in designing systems that scaffold productive cognitive work by leveraging this student expressiveness.

Figures (4)

• Figure 1: Study and Experiment Design
• Figure 2: Comparing Input Characteristics Across Two Conditions
• Figure 3: Plagiarism Scores of Final Output across Two Conditions
• Figure 4: Scores Across Seven Dimensions Compared By Condition and Learning Stage