Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Impact of Multimodal and Conversational AI on Learning Outcomes and Experience

Karan Taneja, Anjali Singh, Ashok K. Goel

Abstract

Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.

Impact of Multimodal and Conversational AI on Learning Outcomes and Experience

Abstract

Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.

Paper Structure

This paper contains 25 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. MuDoC, TexDoC, and DocSearch
  4. Preprocessing and Response Generation in MuDoC
  5. Document Preprocessing:
  6. Response Generation:
  7. Reasoning:
  8. Actions:
  9. Text and Image Retrieval:
  10. Interleaved Grounded Generation:
  11. Pedagogically-aligned Prompt:
  12. Response Post-processing and Rendering:
  13. TexDoC and DocSearch
  14. Experimental Methods
  15. Results
  16. ...and 10 more sections

Figures (3)

  • Figure 1: MuDoC uses GPT-4.1 for Reason-and-Act (ReAct) loop with 4 possible actions viz. Initial Search, Content Search, Confirm Intent, and Final Response.
  • Figure 2: Snapshots from self-learning session: 'Chat' tab shows the conversational AI interface (MuDoC and TexDoC). 'Objectives' tab contains the learning objectives with checkboxes to track progress. 'Document' tab shows the source document for all AI systems. A 'Notepad' box is present on the right for note-taking.
  • Figure 3: Perceived Learning Experience with MuDoC, TexDoC and DocSearch.