Codesigning Ripplet: an LLM-Assisted Assessment Authoring System Grounded in a Conceptual Model of Teachers' Workflows

TL;DR

A conceptual model is developed that characterizes the iterative dual process where teachers develop assessments while simultaneously refining requirements and built Ripplet, a web-based tool with multilevel reusable interactions to support assessment authoring.

Abstract

Assessments are critical in education, but creating them can be difficult. To address this challenge in a grounded way, we partnered with 13 teachers in a seven-month codesign process. We developed a conceptual model that characterizes the iterative dual process where teachers develop assessments while simultaneously refining requirements. To enact this model in practice, we built Ripplet, a web-based tool with multilevel reusable interactions to support assessment authoring. The extended codesign revealed that Ripplet enabled teachers to create formative assessments they would not have otherwise made, shifted their practices from generation to curation, and helped them reflect more on assessment quality. In a user study with 15 additional teachers, compared to their current practices, teachers felt the results were more worth their effort and that assessment quality improved.

Figures (8)

• Figure 1: Overview of the three-phase codesign process and the controlled user study.
• Figure 2: Our conceptual model for teachers' processes of assessment authoring. In this model, a teacher holds some initial ideas about the assessment requirements. They then start and move between the stages to simultaneously develop the assessment and refine assessment requirements, while considering various inputs.
• Figure 3: The main interface (after double-clicking the question stem) of Ripplet consists of four major components: menu bar, search and import bar, question cards, and edit command panel. The menu bar allows users to add questions and navigate, restructure, and export assessments. The search and import bar allows users to look for questions from other assessments in Ripplet and add to the current assessment. Below and lives the questions in the assessment. Ripplet supports two formats of questions: multiple-choice and free-response. Each question is displayed as a question card which has six sections: question stem, answer options (multiple-choice) / answer (free-response), explanation (multiple-choice) / suggested rubric (free-response), topics, skills, and difficulty label. Users can edit a part of a question manually or with ai. A preview window is shown for content that contains LaTeX. User can also edit the entire question with ai. The five icons allow users to manage individual question and create additional ones: using ai to generate similar questions, duplicating, copying to clipboard, undoing an edit, and deletion. Users can also add questions . On the left side is the edit command panel . This panel stores the edit commands a user has made to the questions. Each command has a scope tag that specifies the part of the question it will change once applied.
• Figure 4: Each command can be tagged with specific parts of a question to ensure that only those parts are modified when the command is applied.
• Figure 5: Multilevel edit in Ripplet. Teachers can revise either a specific part (top) or the entire question (bottom) with ai. Differences are shown inline or side-by-side for quick review.