Unlocking Scientific Concepts: How Effective Are LLM-Generated Analogies for Student Understanding and Classroom Practice?

TL;DR

This work tackles how effective LLM-generated analogies are for helping students understand scientific concepts and how teachers can best use them in practice. It employs a two-stage design: a controlled in-class test with high school freshmen to assess unsupervised, student-led understanding, followed by teacher interviews and a one-week classroom field study to gauge classroom applicability and refinement needs. The authors also develop and evaluate a practical system that assists teachers in generating, refining, and managing analogies for instruction. Key findings show that LLM-generated analogies can boost biology understanding but may induce overconfidence and are not reliably beneficial for unsupervised self-learning; in classroom settings, teachers can effectively refine and leverage analogies to improve both in-class learning and homework performance, motivating the development of teacher-centric AI-assisted tools. The work contributes empirical evidence and a practical system that supports analogy-based teaching, along with design implications for future integration of LLMs into education while highlighting subject differences and the need for safeguards against over-reliance and misconceptions.

Abstract

Teaching scientific concepts is essential but challenging, and analogies help students connect new concepts to familiar ideas. Advancements in large language models (LLMs) enable generating analogies, yet their effectiveness in education remains underexplored. In this paper, we first conducted a two-stage study involving high school students and teachers to assess the effectiveness of LLM-generated analogies in biology and physics through a controlled in-class test and a classroom field study. Test results suggested that LLM-generated analogies could enhance student understanding particularly in biology, but require teachers' guidance to prevent over-reliance and overconfidence. Classroom experiments suggested that teachers could refine LLM-generated analogies to their satisfaction and inspire new analogies from generated ones, encouraged by positive classroom feedback and homework performance boosts. Based on findings, we developed and evaluated a practical system to help teachers generate and refine teaching analogies. We discussed future directions for developing and evaluating LLM-supported teaching and learning by analogy.

Figures (7)

• Figure 1: Our Study I address RQ1 through an in-class test. Study II address RQ2 through a pre-class interview and RQ3 through a one-week classroom study. Studies' findings lead us to build an LLM-supported practical system for analogical education.
• Figure 2: The pipeline for analogy generation to explain scientific concepts for data preparation in Study I.
• Figure 3: The examples of final analogies generated from GPT-4o after iterative generation and annotation in Study I.
• Figure 4: Boxplots showing the distribution of answer accuracy and confidence ratings for the two groups and comparing the overall test results and the two subjects. "M" represents Mean, "p" represents the significance of the association between accuracy and group, determined by the exact Wilcoxon-Mann-Whitney test, and * represents significance ($p$<0.05).
• Figure 5: Heatmap of accuracy differences between Group L and Group T for individual questions, with blue indicating higher accuracy for Group L and red for Group T. Each cell contains a bar chart of the respective accuracies and a p-value representing the significance of the association between accuracy and group, determined by Fisher's exact test. For two of the questions, since all students in both groups answered them correctly, Fisher's exact test is not applicable, and $p$ = N/A.