Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course

Amarda Shehu, Adonyas Ababu, Asma Akbary, Griffin Allen, Aroush Baig, Tereana Battle, Elias Beall, Christopher Byrom, Matt Dean, Kate Demarco, Ethan Douglass, Luis Granados, Layla Hantush, Andy Hay, Eleanor Hay, Caleb Jackson, Jaewon Jang, Carter Jones, Quanyang Li, Adrian Lopez, Logan Massimo, Garrett McMullin, Ariana Mendoza Maldonado, Eman Mirza, Hadiya Muddasar, Sara Nuwayhid, Brandon Pak, Ashley Petty, Dryden Rancourt, Lily Rodriguez, Corbin Rogers, Jacob Schiek, Taeseo Seok, Aarav Sethi, Giovanni Vitela, Winston Williams, Jagan Yetukuri

TL;DR

This paper investigates whether consumer chatbots can be said to reason by embedding a field-based, student-led midterm in a general-education AI-literacy course. Students designed prompts across ARC-inspired reasoning categories, executed controlled cross-model comparisons, and separately evaluated answer correctness and explanation grounding. Findings show high performance on short, structured tasks for some models but substantial variability across teams and notable gaps where correct answers were accompanied by flawed explanations. pedagogically, the project demonstrates that AI literacy can be taught as experimental practice, with evaluation skills, prompt design, and interpretability anchored in authentic end-user interactions. The work also highlights limitations of field-style assessments and argues for integrating explanations and prompt-sensitivity into AI evaluation to support informed, critical use of consumer chatbots.

Abstract

Claims about whether large language model (LLM) chatbots "reason" are typically debated using curated benchmarks and laboratory-style evaluation protocols. This paper offers a complementary perspective: a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University, a Mason Core course designed for undergraduates across disciplines with no expected prior STEM exposure. Student teams designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the chatbot's stated reasoning (for example, cases where an answer is correct but the explanation is not, or vice versa). Across eight teams that reported standardized scores, students contributed 80 original reasoning prompts spanning six categories: pattern completion, transformation rules, spatial/visual reasoning, quantitative reasoning, relational/logic reasoning, and analogical reasoning. These prompts yielded 320 model responses plus follow-up explanations. Aggregating team-level results, OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%); explanation validity showed a similar ordering (81.2%, 80.0%, 77.5%, 66.2%). Qualitatively, teams converged on a consistent error signature: strong performance on short, structured math and pattern items but reduced reliability on spatial/visual reasoning and multi-step transformations, with frequent "sound right but reason wrong" explanations. The assignment's primary contribution is pedagogical: it operationalizes AI literacy as experimental practice (prompt design, measurement, rater disagreement, and interpretability/grounding) while producing a reusable, student-generated corpus of reasoning probes grounded in authentic end-user interaction.

Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course

TL;DR

This paper investigates whether consumer chatbots can be said to reason by embedding a field-based, student-led midterm in a general-education AI-literacy course. Students designed prompts across ARC-inspired reasoning categories, executed controlled cross-model comparisons, and separately evaluated answer correctness and explanation grounding. Findings show high performance on short, structured tasks for some models but substantial variability across teams and notable gaps where correct answers were accompanied by flawed explanations. pedagogically, the project demonstrates that AI literacy can be taught as experimental practice, with evaluation skills, prompt design, and interpretability anchored in authentic end-user interactions. The work also highlights limitations of field-style assessments and argues for integrating explanations and prompt-sensitivity into AI evaluation to support informed, critical use of consumer chatbots.

Abstract

Claims about whether large language model (LLM) chatbots "reason" are typically debated using curated benchmarks and laboratory-style evaluation protocols. This paper offers a complementary perspective: a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University, a Mason Core course designed for undergraduates across disciplines with no expected prior STEM exposure. Student teams designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the chatbot's stated reasoning (for example, cases where an answer is correct but the explanation is not, or vice versa). Across eight teams that reported standardized scores, students contributed 80 original reasoning prompts spanning six categories: pattern completion, transformation rules, spatial/visual reasoning, quantitative reasoning, relational/logic reasoning, and analogical reasoning. These prompts yielded 320 model responses plus follow-up explanations. Aggregating team-level results, OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%); explanation validity showed a similar ordering (81.2%, 80.0%, 77.5%, 66.2%). Qualitatively, teams converged on a consistent error signature: strong performance on short, structured math and pattern items but reduced reliability on spatial/visual reasoning and multi-step transformations, with frequent "sound right but reason wrong" explanations. The assignment's primary contribution is pedagogical: it operationalizes AI literacy as experimental practice (prompt design, measurement, rater disagreement, and interpretability/grounding) while producing a reusable, student-generated corpus of reasoning probes grounded in authentic end-user interaction.
Paper Structure (85 sections, 1 figure, 3 tables)

This paper contains 85 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Course Context and Learning Goals
  4. In-class "AI Studios" as the Enabling Course Structure
  5. Mason Core Alignment: AI Literacy as General Education
  6. Why a Chatbot Reasoning Midterm?
  7. Learning Objectives Targeted by the Midterm
  8. Discovery-based Learning: Turning Evaluation into Lived Experience
  9. Midterm project design and specification
  10. Scaffolding Prompt Design: Taxonomy of Reasoning Tasks
  11. Protocol: Controlled Comparison with Explanation Elicitation
  12. Team Structure as "Force Multiplication"
  13. Deliverables: Written Report and Class Presentation as Research-style Artifacts
  14. An Instructor-provided Example: From Toy Patterns to Adversarial Variants
  15. Data and methods
  16. ...and 70 more sections

Figures (1)

  • Figure 1: Consumer chatbot performance aggregated across QC-eligible teams (mean $\pm$ sd): OpenAI GPT 5, xAI Grok 4, Anthropic Claude 4, and Perplexity.