Evaluating the Effect of Pretesting with Conversational AI on Retention of Needed Information

TL;DR

The study addresses whether a pretesting phase before using conversational AI (ChatGPT) enhances retention and transfer of chi-square concepts in a digital learning setting. It uses a true randomized design with two phases: Phase 1 teaches one-way chi-square, followed by Phase 2 that requires two-way chi-square, with pretest or immediate access to ChatGPT. Recall showed no significant group difference, but the pretest group demonstrated a robust improvement on the Final (Transfer) Test ($p<0.001$, $d=8.59$). This indicates that pretesting primes memory and supports deeper engagement even when AI tutoring is available, offering practical guidance for integrating retrieval-based strategies with AI-enabled learning. The work also highlights future research on generalizability, long-term retention, and tailoring pretesting to individual learners in rapid AI-augmented education contexts.

Abstract

This study explores the role of pretesting when integrated with conversational AI tools, specifically ChatGPT, in enhancing learning outcomes. Drawing on existing research, which demonstrates the benefits of pretesting in memory activation and retention, this experiment extends these insights into the context of digital learning environments. A randomized true experimental study was utilized. Participants were divided into two groups: one engaged in pretesting before using ChatGPT for a problem-solving task involving chi-square analysis, while the control group accessed ChatGPT immediately. The results indicate that the pretest group significantly outperformed the no-pretest group in a subsequent test, which suggests that pretesting enhances the retention of complex material. This study contributes to the field by demonstrating that pretesting can augment the learning process in technology-assisted environments by preparing the memory and promoting active engagement with the material. The findings also suggest that learning strategies like pretesting retain their relevance in the context of rapidly evolving AI technologies. Further research and practical implications are presented.

Figures (7)

• Figure 1: Experimental procedure
• Figure 2: Scenario used in Phase 1
• Figure 3: Scenario and two of the items used in the recall test.
• Figure 4: Scenario used in Phase 2 and pretest questions
• Figure 5: Preliminary step and first two steps with prompts and comprehensive questions used in Phase 2 activity.