Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

Mohi Reza, Ioannis Anastasopoulos, Shreya Bhandari, Zachary A. Pardos

TL;DR

The results elucidate the prompt iteration process and validate the tool’s usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.

Abstract

Involving subject matter experts in prompt engineering can guide LLM outputs toward more helpful, accurate, and tailored content that meets the diverse needs of different domains. However, iterating towards effective prompts can be challenging without adequate interface support for systematic experimentation within specific task contexts. In this work, we introduce PromptHive, a collaborative interface for prompt authoring, designed to better connect domain knowledge with prompt engineering through features that encourage rapid iteration on prompt variations. We conducted an evaluation study with ten subject matter experts in math and validated our design through two collaborative prompt-writing sessions and a learning gain study with 358 learners. Our results elucidate the prompt iteration process and validate the tool's usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.

PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

TL;DR

The results elucidate the prompt iteration process and validate the tool’s usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.

Abstract

Involving subject matter experts in prompt engineering can guide LLM outputs toward more helpful, accurate, and tailored content that meets the diverse needs of different domains. However, iterating towards effective prompts can be challenging without adequate interface support for systematic experimentation within specific task contexts. In this work, we introduce PromptHive, a collaborative interface for prompt authoring, designed to better connect domain knowledge with prompt engineering through features that encourage rapid iteration on prompt variations. We conducted an evaluation study with ten subject matter experts in math and validated our design through two collaborative prompt-writing sessions and a learning gain study with 358 learners. Our results elucidate the prompt iteration process and validate the tool's usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.

Paper Structure

This paper contains 33 sections, 1 equation, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Integrating Subject Matter Expertise into Prompt Engineering
  4. Usability Challenges of Designing Prompt Engineering Interfaces
  5. Designing Effective Content Authoring Tools for Intelligent Tutoring Systems
  6. Generative AI in Tutoring Systems
  7. Designing PromptHive
  8. Understanding the Expert Workflow
  9. Eliciting Design Requirements
  10. Brainstorming & Design Review Sessions
  11. Developing the PromptHive Interface
  12. Study 1: User Evaluation with Subject Matter Experts
  13. Research Questions
  14. Participants
  15. Procedure & Tasks
  16. ...and 18 more sections

Figures (13)

  • Figure 1: The subject matter expert workflow for manually authoring content in OATutor.
  • Figure 2: How the interface elements in PromptHive map to the 4-stage, 2-level workflow.
  • Figure 3: NASA-TLX ratings for the content authoring workflow in PromptHive versus manual.
  • Figure 4: Participant rating distribution for the Explainable AI (XAI) Trust scale adapted from Hoffman et al. hoffman2023measures
  • Figure 5: Participants' influence on each other's prompts. Outer border colors for lessons correspond to lessons assigned to participants, with the same color representing the same participant. The fill color of lesson numbers and the arrows indicate the textbook-level source for the lesson prompts.
  • ...and 8 more figures