Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback

Stanislav Pozdniakov, Jonathan Brazil, Solmaz Abdi, Aneesha Bakharia, Shazia Sadiq, Dragan Gasevic, Paul Denny, Hassan Khosravi

TL;DR

The paper argues that GenAI and LLMs hold promise for enhancing education but that prevalent CUI-based use suffers from AI-literacy, privacy, and governance challenges. It presents a two-component design framework—application design and interaction design—that aims to reduce AI-proficiency barriers and place educator oversight at the center of workflow. The framework is instantiated in Feedback Copilot, a tool that generates personalized open-response feedback and autonomously flags outputs that need human review according to predefined criteria. An empirical study with 338 students shows that an advanced, role-informed prompting variant improves feedback quality, while lower-performing students tend to receive lower-quality feedback unless instructors oversee and intervene. The work offers practical guidance for researchers, educators, and technologists on building ethical, effective GenAI-enabled educational tools that augment, rather than replace, human instruction.

Abstract

Incorporating Generative AI (GenAI) and Large Language Models (LLMs) in education can enhance teaching efficiency and enrich student learning. Current LLM usage involves conversational user interfaces (CUIs) for tasks like generating materials or providing feedback. However, this presents challenges including the need for educator expertise in AI and CUIs, ethical concerns with high-stakes decisions, and privacy risks. CUIs also struggle with complex tasks. To address these, we propose transitioning from CUIs to user-friendly applications leveraging LLMs via API calls. We present a framework for ethically incorporating GenAI into educational tools and demonstrate its application in our tool, Feedback Copilot, which provides personalized feedback on student assignments. Our evaluation shows the effectiveness of this approach, with implications for GenAI researchers, educators, and technologists. This work charts a course for the future of GenAI in education.

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback

TL;DR

The paper argues that GenAI and LLMs hold promise for enhancing education but that prevalent CUI-based use suffers from AI-literacy, privacy, and governance challenges. It presents a two-component design framework—application design and interaction design—that aims to reduce AI-proficiency barriers and place educator oversight at the center of workflow. The framework is instantiated in Feedback Copilot, a tool that generates personalized open-response feedback and autonomously flags outputs that need human review according to predefined criteria. An empirical study with 338 students shows that an advanced, role-informed prompting variant improves feedback quality, while lower-performing students tend to receive lower-quality feedback unless instructors oversee and intervene. The work offers practical guidance for researchers, educators, and technologists on building ethical, effective GenAI-enabled educational tools that augment, rather than replace, human instruction.

Abstract

Incorporating Generative AI (GenAI) and Large Language Models (LLMs) in education can enhance teaching efficiency and enrich student learning. Current LLM usage involves conversational user interfaces (CUIs) for tasks like generating materials or providing feedback. However, this presents challenges including the need for educator expertise in AI and CUIs, ethical concerns with high-stakes decisions, and privacy risks. CUIs also struggle with complex tasks. To address these, we propose transitioning from CUIs to user-friendly applications leveraging LLMs via API calls. We present a framework for ethically incorporating GenAI into educational tools and demonstrate its application in our tool, Feedback Copilot, which provides personalized feedback on student assignments. Our evaluation shows the effectiveness of this approach, with implications for GenAI researchers, educators, and technologists. This work charts a course for the future of GenAI in education.
Paper Structure (43 sections, 11 figures, 1 table)

This paper contains 43 sections, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. The Application of Generative AI in Education
  4. The Foundations of Automated Feedback
  5. LLMs for Automated Feedback
  6. GenAI and LLM Interfaces for Guidance and Prompt Engineering Support
  7. Research Gap and Contribution
  8. Empowering Educators through a Generative AI Framework
  9. Application Design
  10. 1-Educational Task.
  11. 2-Pedagogical Framework.
  12. 3-Evaluation Criteria.
  13. 4-Data, GenAI Model, and Prompting Template.
  14. Interaction Design
  15. 5-Interface Design.
  16. ...and 28 more sections

Figures (11)

  • Figure 1: The framework, formulated to aid the development of GenAI applications for educational tasks, consists of two components. Component one (Steps 1-4) guides the application design and high-level interface considerations, starting with the selection of the educational task and GenAI model, and ending with the creation of a user interface for GenAI model inputs. Component two (Steps 5-8) guides user interaction design, including steps for interactive intent alignment with the GenAI, prompt preview, active evaluation criteria selection, and GenAI model output preview with evaluation results.
  • Figure 2: This figure illustrates the framework's instantiation, demonstrating its application to inform Feedback Copilot's development. Steps 1-4 demonstrate the potential alternatives for GenAI application development for feedback tasks. Steps 5-8 show possible design decisions for UIs for instructor input, prompt pipeline interactions, evaluation pipeline, and generated feedback output.
  • Figure 3: This figure provides an overview of the Feedback Copilot capabilities, enabling instructors to overview the status of the feedback generation (A) and action on the feedback (B). Each view is marked with a numbered overlay box, indicating a mapping to the corresponding step in the proposed framework.
  • Figure 4: The figure shows the Feedback Copilot interface. The instructor starts with specifying inputs to generate personalized feedback (A). Then, the instructor specifies feedback criteria (B). Feedback preview (C) provides a glimpse of a sample of generated feedback. Each view is numbered, mapping to the framework step.
  • Figure 5: The figure provides an overview of the feedback generated for multiple students or student groups in the course, showing specific and overall feedback evaluation (A). Additional information about students' backgrounds or assignment results could be included (B). It enables instructors to drill-down into individual overviews of the generated feedback that instructors can review (C). Each view is marked with a numbered overlay box, indicating a mapping to the corresponding step in the proposed framework.
  • ...and 6 more figures