Investigating Middle School Students Question-Asking and Answer-Evaluation Skills When Using ChatGPT for Science Investigation

TL;DR

This study investigates how 14–15-year-old students use ChatGPT to solve science problems, focusing on their ability to formulate efficient prompts and critically evaluate AI outputs. Using six tasks with randomized prompt quality and comprehensive questionnaires, the authors show that middle-schoolers exhibit only modest discrimination of prompt quality, struggle to evaluate AI-provided answers, and produce learning outcomes that depend on the quality of AI responses and students’ follow-up actions. Learning is best predicted by the frequency of satisfactory AI answers, sensitivity to answer quality, and the propensity to ask follow-up questions when unsatisfied, with metacognition positively moderating prompting effectiveness. The findings highlight the need for targeted AI-literacy training and metacognitive scaffolding in middle school to foster epistemic vigilance and effective GenAI-assisted learning in science education.

Abstract

Generative AI (GenAI) tools such as ChatGPT allow users, including school students without prior AI expertise, to explore and address a wide range of tasks. Surveys show that most students aged eleven and older already use these tools for school-related activities. However, little is known about how they actually use GenAI and how it impacts their learning. This study addresses this gap by examining middle school students ability to ask effective questions and critically evaluate ChatGPT responses, two essential skills for active learning and productive interactions with GenAI. 63 students aged 14 to 15 were tasked with solving science investigation problems using ChatGPT. We analyzed their interactions with the model, as well as their resulting learning outcomes. Findings show that students often over-relied on ChatGPT in both the question-asking and answer-evaluation phases. Many struggled to use clear questions aligned with task goals and had difficulty judging the quality of responses or knowing when to seek clarification. As a result, their learning performance remained moderate: their explanations of the scientific concepts tended to be vague, incomplete, or inaccurate, even after unrestricted use of ChatGPT. This pattern held even in domains where students reported strong prior knowledge. Furthermore, students self-reported understanding and use of ChatGPT were negatively associated with their ability to select effective questions and evaluate responses, suggesting misconceptions about the tool and its limitations. In contrast, higher metacognitive skills were positively linked to better QA-related skills. These findings underscore the need for educational interventions that promote AI literacy and foster question-asking strategies to support effective learning with GenAI.

Figures (6)

• Figure 1: An example of the tasks proposed. Each task includes specific informational elements presented in both text and image format to understand its specific context, along with a text-based instruction. (A) is an example for a case with a 'non-efficient' question for the prompt and (B) is for an 'efficient' one.
• Figure 2: Overall study timeline
• Figure 3: Students' sensitivity to the suggested prompts' quality and abilities to self-generate efficient prompts.
• Figure 4: Students' performance in evaluating ChatGPT's answers and link with prior domain knowledge knowledge.
• Figure 5: Students, with full access to ChatGPT, had an average chance-level success rate over the six problems.