Decomposed Prompting to Answer Questions on a Course Discussion Board

TL;DR

This paper tackles the risk of incorrect answers in course discussion-board Q&A by proposing a decomposed prompting framework that uses a Mixture of Experts to first classify questions into four types: conceptual, homework, logistics, and not answerable. Depending on the type, a specialized prompting strategy is used to generate answers or to defer to instructors, enabling risk-aware deployment. The authors demonstrate 81% accuracy in question classification and provide an analysis showing conceptual questions are easier for the LLM to answer than homework or logistics questions, with human evaluation revealing limited quality for conceptual answers and several failure modes. The work highlights a modular, controllable approach to deploying LLMs in education and suggests future improvements through fine-tuning on course data and integrating semantic processing for more robust handling of different question types.

Abstract

We propose and evaluate a question-answering system that uses decomposed prompting to classify and answer student questions on a course discussion board. Our system uses a large language model (LLM) to classify questions into one of four types: conceptual, homework, logistics, and not answerable. This enables us to employ a different strategy for answering questions that fall under different types. Using a variant of GPT-3, we achieve $81\%$ classification accuracy. We discuss our system's performance on answering conceptual questions from a machine learning course and various failure modes.

Figures (2)

• Figure 1: Mixture of Experts for answering questions using decomposed prompting.
• Figure 2: Prompt used for classification (left) and conceptual Q&A (right).