Easy Come, Easy Go? Examining the Perceptions and Learning Effects of LLM-based Chatbot in the Context of Search-as-Learning

TL;DR

This study systematically evaluates LLM-based chatbots as a SAL tool against textbooks and web search across perceptions, throughput, and retention. Using a large-scale survey plus a within-subject experiment (N=92), it finds that chatbots boost immediate comprehension and information throughput but do not yield superior long-term retention compared with traditional modalities. The results reveal a pedagogical tension: learners favor efficiency, while educators emphasize cognitive effort for durable knowledge, suggesting that effective SAL design should blend AI-assisted rapid encoding with deliberate reflection and discussion. The findings inform design guidelines for human-AI collaborative learning that optimize short-term gains while preserving long-term understanding. The work highlights the need for scaffolding and blended approaches to harness AI while mitigating shallow processing and hallucination risks.

Abstract

The cognitive process of Search-as-Learning (SAL) is most effective when searching promotes active encoding of information. The rise of LLMs-based chatbots, which provide instant answers, introduces a trade-off between efficiency and depth of processing. Such answer-centric approaches accelerate information access, but they also raise concerns about shallower learning. To examine these issues in the context of SAL, we conducted a large-scale survey of educators and students to capture perceived risks and benefits of LLM-based chatbots. In addition, we adopted the encoding-storage paradigm to design a within-subjects experiment, where participants (N=92) engaged in SAL tasks using three different modalities: books, search engines, and chatbots. Our findings provide a counterintuitive insight into stakeholder concerns: while LLM-based chatbots and search engines validated perceived benefits on learning efficiency by outperforming book-based search in immediate conceptual understanding, they did not result in a long-term inferiority as feared. Our study provides insights for designing human-AI collaborative learning systems that promote cognitive engagement by balancing learning efficiency and long-term knowledge retention.

Figures (9)

• Figure 1: Procedure over the five steps in a mixed-method within-subject study with 92 university students.
• Figure 2: Overview of the SAL logger of [toolboxstyle, background-color=webcolor]Search Engine. The interfaces for [toolboxstyle, background-color=bookcolor]Book and [toolboxstyle, background-color=chatbotcolor]Chatbot follow a similar structure, consisting of two interface components: a the content panel and b the note panel. A Chrome extension plug-in c was provided for the [toolboxstyle, background-color=bookcolor]Book condition to access pre-selected textbooks in PDF format. The experiment starts by inputting information such as participant ID, session (from Day 1 to Day 3), experiment condition, and study subject on top of the note panel. Participants start by initiating a question themselves in the Question section. As they engage in self-guided search and learning on the content panel, they can drag relevant information or organize it into their own words in the Note section. Once they find an answer to their query, participants complete the Answer section and submit it. They can add new notes by clicking the ’+’ button.
• Figure 3: Despite a shared learning objective, students' engagement patterns, interaction types, and the resultant cognitive burdens can diverge significantly across learning tools. By comparatively analyzing different SAL tools: books, search engines, and chatbots, we aim to clarify chatbots' unique contributions and limitations, offering insights into better LLM support in SAL practices. (The images are from actual experiment logs: book search highlight of S88, search engine query of S69, and chatbot log of S36.)
• Figure 4: Bar chart showing mean preference ranks from educator and student survey responses. A lower mean rank (↓) indicates higher preference (i.e., 1 = most preferred). The left plot (a) presents educators’ rankings of the three SAL tools, and the right plot (b) shows students’ rankings. Significance is based on a Friedman test and marked as $p$ < .001(***).
• Figure 5: Bar chart showing the number of search Q&A pairs (left) and the average time spent per Q&A pair (right). ANOVA was conducted to compare information throughput across the conditions. Significance is marked as $p$ < 0.1 ($^{+}$), $p$ < 0.05 (*), $p$ < 0.01 (**), or $p$ < 0.001 (***).