Changing the Optics: Comparing Traditional and Retrieval-Augmented GenAI E-Tutorials in Interdisciplinary Learning

TL;DR

Traditional users maintained greater awareness and focus of the information space, whereas GenAI users exhibited more proactive and exploratory behaviors with lower cognitive load due to the querying-driven interaction.

Abstract

Understanding information-seeking behaviors in e-learning is critical, as learners must often make sense of complex and fragmented information, a challenge compounded in interdisciplinary fields with diverse prior knowledge. Compared to traditional e-tutorials, GenAI e-tutorials offer new ways to navigate information spaces, yet how they shape learners information-seeking behaviors remains unclear. To address this gap, we characterized behavioral differences between traditional and GenAI-mediated e-tutorial learning using the three search modes of orienteering. We conducted a between-subject study in which learners engaged with either a traditional e-tutorial or a GenAI e-tutorial accessing the same underlying information content. We found that the traditional users maintained greater awareness and focus of the information space, whereas GenAI users exhibited more proactive and exploratory behaviors with lower cognitive load due to the querying-driven interaction. These findings offer guidance for designing tutorials in e-learning.

Figures (7)

• Figure 1: Conceptual Depiction of Orienteering o1993orienteering. Users transition between states toward the finish line in the question-answering task by relying on either a traditional or a retrieval-augmented GenAI e-tutorial. Both tutorials provide access to the same information space, but differences in how the information is accessed distinguish them. This process of seeking information is analogous to navigation in a forest or fog, as users proceed without full knowledge of the underlying rules or state space.
• Figure 2: Study Overview: (A) Retrieval-Augmented Generation system diagram; (B) comparison of information-seeking behavior between a control group using a traditional e-tutorial and a treatment group using a retrieval-augmented large language model (hereafter GenAI) e-tutorial; (C) in-person survey and observation workflow; and (D) online interview workflow.
• Figure S1: Model Selection for the Generator. To minimize potential model bias, we selected a model with limited prior domain-specific knowledge of HyPhy kosakovsky2020hyphy. Task-relevant and task-irrelevant user queries were combined to evaluate three well-known large language models groq_models_docsmeta_llama3.3_modelcardagarwal2025gpt. Model performance was assessed both (A) without and (B) with retrieval augmentation. The top three retrieved chunks list shows partial document fragments returned based on their similarity to the user query. Because similarity is estimated using L2 (Euclidean) distance, a lower distance indicates higher relevance. The 'nomic-embed-text' model ollama_nomic_embed_text was used for embedding. The excerpts from the retrieved chunks and GenAI responses shown in the figure highlight limitations of casual input augmentation in GenAI. Incorrect information about HyPhy is highlighted in red. Based on the findings, the 'llama-3.3-70b-versatile' model meta_llama3.3_modelcard was selected, the data analysis plan was revised and the distance threshold was updated to 0.9 RA-BSTS_repo.
• Figure S2: In-Person Experimental Setup: (A) a private room with a door and privacy glass that all in-person experiments were conducted; (B) the default computer screen; (C) the traditional e-tutorial screen; and (D) the GenAI e-tutorial screen.
• Figure S3: Video Version of the Task Scenario. An actor played dual roles of (A) a student researcher (the participant) and (B) a mentor in this point-of-view (POV) video. There were two versions of the video: (C) one that included the screen of the traditional e-tutorial, and (D) another that included the screen of the GenAI e-tutorial. Link to the video version of the task scenario youtube_shorts_Bnxgrbpujrk: https://youtube.com/shorts/Bnxgrbpujrk.