Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Bespoke Bots": Diverse Instructor Needs for Customizing Generative AI Classroom Chatbots

Irene Hou, Zeyu Xiong, Philip J. Guo, April Yi Wang

TL;DR

It is found that instructors consistently prioritized the ability to customize chatbot behavior to align with course materials and pedagogical strategies and de-prioritized customizing persona/tone, suggesting that modular AI chatbots may provide a promising path forward.

Abstract

Instructors are increasingly experimenting with AI chatbots for classroom support. To investigate how instructors adapt chatbots to their own contexts, we first analyzed existing resources that provide prompts for educational purposes. We identified ten common categories of customization, such as persona, guardrails, and personalization. We then conducted interviews with ten university STEM instructors and asked them to card-sort the categories into priorities. We found that instructors consistently prioritized the ability to customize chatbot behavior to align with course materials and pedagogical strategies and de-prioritized customizing persona/tone. However, their prioritization of other categories varied significantly by course size, discipline, and teaching style, even across courses taught by the same individual, highlighting that no single design can meet all contexts. These findings suggest that modular AI chatbots may provide a promising path forward. We offer design implications for educational developers building the next generation of customizable classroom AI systems.

"Bespoke Bots": Diverse Instructor Needs for Customizing Generative AI Classroom Chatbots

TL;DR

It is found that instructors consistently prioritized the ability to customize chatbot behavior to align with course materials and pedagogical strategies and de-prioritized customizing persona/tone, suggesting that modular AI chatbots may provide a promising path forward.

Abstract

Instructors are increasingly experimenting with AI chatbots for classroom support. To investigate how instructors adapt chatbots to their own contexts, we first analyzed existing resources that provide prompts for educational purposes. We identified ten common categories of customization, such as persona, guardrails, and personalization. We then conducted interviews with ten university STEM instructors and asked them to card-sort the categories into priorities. We found that instructors consistently prioritized the ability to customize chatbot behavior to align with course materials and pedagogical strategies and de-prioritized customizing persona/tone. However, their prioritization of other categories varied significantly by course size, discipline, and teaching style, even across courses taught by the same individual, highlighting that no single design can meet all contexts. These findings suggest that modular AI chatbots may provide a promising path forward. We offer design implications for educational developers building the next generation of customizable classroom AI systems.
Paper Structure (26 sections, 1 figure, 3 tables)

This paper contains 26 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Instructors and the Customization of Classroom Technologies
  4. Generative AI and Chatbots in Education
  5. Content Analysis of Existing Prompts and Resources
  6. Content Analysis Methods
  7. Content Analysis Results
  8. Card-Sorting Interviews with Instructors
  9. Participants
  10. Interview Protocol and Analysis
  11. Study Scope and Limitations
  12. Results
  13. Where instructors converged on priorities
  14. Instructors wanted custom chatbots to have access to course-specific materials.
  15. Instructors wanted to customize chatbot pedagogical strategy.
  16. ...and 11 more sections

Figures (1)

  • Figure 1: P6's finalized card-sorting interview artifact, with cards sorted into the "high priority," "medium priority," and "low priority" buckets. Card-sorting interviews were conducted via a collaborative online whiteboarding tool, with each card depicting a potential area of chatbot customization for instructors and example use cases. Refer to Table \ref{['tab:chatbot-features']} to see detailed card categories and examples.