VizQStudio: Iterative Visualization Literacy MCQs Design with Simulated Students

TL;DR

VizQStudio is presented, a visual analytics system that supports instructors in iteratively designing and refining visualization literacy MCQs using MLLM-powered simulated students and reframes MLLM-based student simulation in assessment authoring as a design-time, exploratory aid.

Abstract

Multiple-choice questions (MCQs) are a widely used educational tool, particularly in domains such as visualization literacy that require broad conceptual coverage and support diverse real-world applications. However, designing high-quality visualization literacy MCQs remains challenging, as instructors must coordinate multimodal elements (e.g., charts, question stems, and distractors), address diverse visualization tasks, and accommodate learners with heterogeneous backgrounds. Existing visualization literacy assessments primarily rely on standardized, fixed item banks, offering limited support for iterative question design that adapts to differences in learners' abilities, backgrounds, and reasoning strategies. To address these challenges, we present VizQStudio, a visual analytics system that supports instructors in iteratively designing and refining visualization literacy MCQs using MLLM-powered simulated students. Instructors can specify diverse student profiles spanning demographics, knowledge levels, and learning-related traits. The system then visualizes how simulated students reason about and respond to different question components, helping instructors explore potential misconceptions, difficulty calibration, and design trade-offs prior to classroom deployment. We investigate VizQStudio through a mixed-method evaluation, including expert interviews, case studies, a classroom deployment, and a large-scale online study. Overall, this work reframes MLLM-based student simulation in assessment authoring as a design-time, exploratory aid. By examining both its value and limitations in realistic instructional settings, we surface design insights that inform how future systems can support instructor-centered, iterative, and responsible uses of AI for multimodal assessment design in visualization literacy and related domains.

Figures (11)

• Figure 1: Overview of the proposed MCQ features and student agent profile settings, with detailed definitions provided in the supplementary. (A) MCQ features are organized into four categories: question, chart, distractor, and knowledge. (B) Agent settings cover demographic attributes, learning traits, and knowledge point relevant to visualization literacy.
• Figure 2: The question generation pipeline. (A) The system analyzes the instructor's input—such as textual descriptions, sample figures, or lecture slides—using a GPT-4o module. It then extracts feature requirements (e.g., chart features, knowledge points, distractor features) in accordance with the feature set described in \ref{['fig:Feature_Details']}(B) These extracted features are used to retrieve the most similar MCQ templates, along with corresponding chart data and code, from an external database we collected. (C) In the first step, the system uses the retrieved D3.js code, data, and QA templates—combined with the corresponding chart image and user-specified feature requirements—to modify the template code and produce a customized chart figure. In the second step, an additional MLLM module processes the generated chart image, the template QA, and the instructor's requirements for question content and distractors to generate the final QA set. This two-step process yields an output MCQ that fully aligns with the instructor's design intent.
• Figure 3: Overview of the iterative MCQ update process. (A) The system starts with the previous MCQ (chart image, code, data, and QA content) as references. (B) Alongside these references, the MLLM module also takes in the user’s revision prompt and selected features from the Feature View. (C) The MLLM then updates the chart code and CSV data, renders the revised chart, and modifies the QA content, producing an updated MCQ aligned with the user's instructions.
• Figure 4: System interface of VizQStudio. The interface is organized into five main panels: (A) Sample View, where users provide a sample question and/or reference figure; (B) Feature View, which displays four feature categories and historical user choices to control and adjust question generation; (C) Simulation View, which classifies the simulated students and presents both the generated question and simulation outcomes; (D) Agent Profile View where users broadly define the characteristics of the student agents; and (E) Agent Setting View, which enables fine-grained control and adjustment of the MLLM-simulated students' profiles. The four side panels (A, B, D, E) serve as user control areas, while the central (C) Simulation View aligns each component of the Generated Question with its corresponding simulated feedback in Simulation Results zone. This layout allows users to immediately see how each round of edits or feature adjustments influences student responses.
• Figure 5: (A) Each color-coded group card displays the cluster size and demographic details. Radar charts illustrate each group's average traits and knowledge levels. Users can interactively drag the radar charts to adjust the group levels. (B) Each stacked bar chart illustrates each student group's feedback on the question stem and related features. Colored bars represent feedback for the current version, while grey bars indicate responses from the previous version for quick comparison.