Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhysicsAssistant: An LLM-Powered Interactive Learning Robot for Physics Lab Investigations

Ehsan Latif, Ramviyas Parasuraman, Xiaoming Zhai

TL;DR

PhysicsAssistant tackles the need for personalized, real-time physics lab support in K-12 by integrating multimodal perception (YOLOv8) with an LLM-driven dialogue system and a rigorous response validation workflow. The system processes speech and visual inputs to generate context-rich prompts, reasons through physics concepts, validates linguistic and domain accuracy, and delivers spoken feedback. In a study with ten eighth-grade students, PhysicsAssistant achieved strong factual accuracy and significantly faster response times than GPT-4, though it lagged in higher-order conceptual and procedural reasoning, indicating a trade-off between speed and depth of understanding. The work demonstrates a practical, scalable approach to offloading repetitive teaching tasks and lays groundwork for improvements in higher-order reasoning and personalization in educational robotics.

Abstract

Robot systems in education can leverage Large language models' (LLMs) natural language understanding capabilities to provide assistance and facilitate learning. This paper proposes a multimodal interactive robot (PhysicsAssistant) built on YOLOv8 object detection, cameras, speech recognition, and chatbot using LLM to provide assistance to students' physics labs. We conduct a user study on ten 8th-grade students to empirically evaluate the performance of PhysicsAssistant with a human expert. The Expert rates the assistants' responses to student queries on a 0-4 scale based on Bloom's taxonomy to provide educational support. We have compared the performance of PhysicsAssistant (YOLOv8+GPT-3.5-turbo) with GPT-4 and found that the human expert rating of both systems for factual understanding is the same. However, the rating of GPT-4 for conceptual and procedural knowledge (3 and 3.2 vs 2.2 and 2.6, respectively) is significantly higher than PhysicsAssistant (p < 0.05). However, the response time of GPT-4 is significantly higher than PhysicsAssistant (3.54 vs 1.64 sec, p < 0.05). Hence, despite the relatively lower response quality of PhysicsAssistant than GPT-4, it has shown potential for being used as a real-time lab assistant to provide timely responses and can offload teachers' labor to assist with repetitive tasks. To the best of our knowledge, this is the first attempt to build such an interactive multimodal robotic assistant for K-12 science (physics) education.

PhysicsAssistant: An LLM-Powered Interactive Learning Robot for Physics Lab Investigations

TL;DR

PhysicsAssistant tackles the need for personalized, real-time physics lab support in K-12 by integrating multimodal perception (YOLOv8) with an LLM-driven dialogue system and a rigorous response validation workflow. The system processes speech and visual inputs to generate context-rich prompts, reasons through physics concepts, validates linguistic and domain accuracy, and delivers spoken feedback. In a study with ten eighth-grade students, PhysicsAssistant achieved strong factual accuracy and significantly faster response times than GPT-4, though it lagged in higher-order conceptual and procedural reasoning, indicating a trade-off between speed and depth of understanding. The work demonstrates a practical, scalable approach to offloading repetitive teaching tasks and lays groundwork for improvements in higher-order reasoning and personalization in educational robotics.

Abstract

Robot systems in education can leverage Large language models' (LLMs) natural language understanding capabilities to provide assistance and facilitate learning. This paper proposes a multimodal interactive robot (PhysicsAssistant) built on YOLOv8 object detection, cameras, speech recognition, and chatbot using LLM to provide assistance to students' physics labs. We conduct a user study on ten 8th-grade students to empirically evaluate the performance of PhysicsAssistant with a human expert. The Expert rates the assistants' responses to student queries on a 0-4 scale based on Bloom's taxonomy to provide educational support. We have compared the performance of PhysicsAssistant (YOLOv8+GPT-3.5-turbo) with GPT-4 and found that the human expert rating of both systems for factual understanding is the same. However, the rating of GPT-4 for conceptual and procedural knowledge (3 and 3.2 vs 2.2 and 2.6, respectively) is significantly higher than PhysicsAssistant (p < 0.05). However, the response time of GPT-4 is significantly higher than PhysicsAssistant (3.54 vs 1.64 sec, p < 0.05). Hence, despite the relatively lower response quality of PhysicsAssistant than GPT-4, it has shown potential for being used as a real-time lab assistant to provide timely responses and can offload teachers' labor to assist with repetitive tasks. To the best of our knowledge, this is the first attempt to build such an interactive multimodal robotic assistant for K-12 science (physics) education.
Paper Structure (29 sections, 5 equations, 3 figures, 2 tables)

This paper contains 29 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed PhysicsAssistant System
  4. Speech-to-Text Encoding
  5. Image Processing
  6. Prompt Designing
  7. LLM Processing
  8. Function $G$ Derivation
  9. Perceiving the Prompt and Providing Human-like Feedback
  10. Example LLM Response
  11. Response Validation
  12. Linguistic and Coherence Checks
  13. Physics-based Verifications
  14. Implementation Details
  15. Response Revision Mechanism
  16. ...and 14 more sections

Figures (3)

  • Figure 1: An overview of LLM-powered PhysicsAssistant supporting students to perform 8th-grade projectile motion experiment.
  • Figure 2: PhysicsAssistant's System Architecture, Robot receives audio and visual instructions using an embedded microphone and camera and invokes internal modules for processing, then speaks the response using the inbuilt speaker to students.
  • Figure 3: Object detection results using YOLOv8 model