Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing Prompt Analytics Dashboards to Analyze Student-ChatGPT Interactions in EFL Writing

Minsun Kim, SeonGyeom Kim, Suyoun Lee, Yoosang Yoon, Junho Myung, Haneul Yoo, Hyunseung Lim, Jieun Han, Yoonsu Kim, So-Yeon Ahn, Juho Kim, Alice Oh, Hwajung Hong, Tak Yeon Lee

TL;DR

A prototype of Prompt Analytics Dashboard (PAD) is developed that integrates the essay editing history and chat logs between students and ChatGPT, and teacher's feedback on the prototype informs additional features and unmet needs for designing future PAD.

Abstract

While ChatGPT has significantly impacted education by offering personalized resources for students, its integration into educational settings poses unprecedented risks, such as inaccuracies and biases in AI-generated content, plagiarism and over-reliance on AI, and privacy and security issues. To help teachers address such risks, we conducted a two-phase iterative design process that comprises surveys, interviews, and prototype demonstration involving six EFL (English as a Foreign Language) teachers, who integrated ChatGPT into semester-long English essay writing classes. Based on the needs identified during the initial survey and interviews, we developed a prototype of Prompt Analytics Dashboard (PAD) that integrates the essay editing history and chat logs between students and ChatGPT. Teacher's feedback on the prototype informs additional features and unmet needs for designing future PAD, which helps them (1) analyze contextual analysis of student behaviors, (2) design an overall learning loop, and (3) develop their teaching skills.

Designing Prompt Analytics Dashboards to Analyze Student-ChatGPT Interactions in EFL Writing

TL;DR

A prototype of Prompt Analytics Dashboard (PAD) is developed that integrates the essay editing history and chat logs between students and ChatGPT, and teacher's feedback on the prototype informs additional features and unmet needs for designing future PAD.

Abstract

While ChatGPT has significantly impacted education by offering personalized resources for students, its integration into educational settings poses unprecedented risks, such as inaccuracies and biases in AI-generated content, plagiarism and over-reliance on AI, and privacy and security issues. To help teachers address such risks, we conducted a two-phase iterative design process that comprises surveys, interviews, and prototype demonstration involving six EFL (English as a Foreign Language) teachers, who integrated ChatGPT into semester-long English essay writing classes. Based on the needs identified during the initial survey and interviews, we developed a prototype of Prompt Analytics Dashboard (PAD) that integrates the essay editing history and chat logs between students and ChatGPT. Teacher's feedback on the prototype informs additional features and unmet needs for designing future PAD, which helps them (1) analyze contextual analysis of student behaviors, (2) design an overall learning loop, and (3) develop their teaching skills.
Paper Structure (41 sections, 3 figures, 2 tables)

This paper contains 41 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Model Systems in Language Education
  4. Human-Centered Learning Analytics
  5. Method
  6. A Semester-long Essay Writing Course with ChatGPT
  7. Participants
  8. Design Process
  9. Needs Finding(Interview 1)
  10. Interview Process
  11. Examining teachers' perspective
  12. Observing how teachers evaluate student-ChatGPT interaction
  13. Gathering design needs
  14. Analysis
  15. Challenges and Needs on Analyzing ChatGPT-Student Interaction
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Contents of interview handouts.
  • Figure 2: Screenshot of dashboard prototype. (a) Population pool: The dashboard displays information calculated based on the selected population pool. Teachers can narrow the analysis pool by selecting classes, students, and weeks. (b) Overview: This is an overview chart with the x-axis fixed to weekly, allowing you to see the chat count over time, the last saved essay score graded by AI, and the misuse count. (c) This area consists of three tabs, (c-1) a chart for chatting patterns, (c-2) additional filtering options based on learning objectives, and (c-3) the ability to deliver messages to students. (d) Teachers can track changes to essays by moving sliders. (e) Shows the students' entire chat history. Teachers can search, and each prompt has a tag so teachers can check the chat content.
  • Figure 3: A dashboard system architecture.