Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch

Evanfiya Logacheva, Arto Hellas, Tsvetomila Mihaylova, Juha Sorva, Ava Heinonen, Juho Leinonen

TL;DR

This work tackles the quality and hallucination risks of AI-generated educational diagrams by introducing an RST-guided in-context learning pipeline with two variants (RST1, RST2) and zero-shot baselines. The approach combines RST analysis, similarity-based exemplar selection, stepwise prompting, and iterative repairs to produce renderable Graphviz diagrams while aiming to reduce factual and faithfulness hallucinations. A five-point rubric evaluated by experts and automated methods shows that RST-based prompts can improve diagram faithfulness and reduce hallucinations, with o3 generally outperforming GPT-4o; however, factual hallucinations persist and context complexity amplifies hallucination risks. The study highlights the need for safeguards and further empirical work to address LLM stochasticity and the propagation of hallucinations across pipeline steps, especially in educational settings.

Abstract

Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models' output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs' stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output.

Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch

TL;DR

This work tackles the quality and hallucination risks of AI-generated educational diagrams by introducing an RST-guided in-context learning pipeline with two variants (RST1, RST2) and zero-shot baselines. The approach combines RST analysis, similarity-based exemplar selection, stepwise prompting, and iterative repairs to produce renderable Graphviz diagrams while aiming to reduce factual and faithfulness hallucinations. A five-point rubric evaluated by experts and automated methods shows that RST-based prompts can improve diagram faithfulness and reduce hallucinations, with o3 generally outperforming GPT-4o; however, factual hallucinations persist and context complexity amplifies hallucination risks. The study highlights the need for safeguards and further empirical work to address LLM stochasticity and the propagation of hallucinations across pipeline steps, especially in educational settings.

Abstract

Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models' output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs' stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output.
Paper Structure (38 sections, 2 equations, 16 figures, 5 tables)

This paper contains 38 sections, 2 equations, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Rhetorical Structure Theory
  4. In-context Learning
  5. Taxonomy of Hallucination in LLM-Generated Content
  6. Proposed Method
  7. High-level Overview
  8. RST-guided Diagram Generation Pipeline Design
  9. Step 1: RST analysis.
  10. Step 2: Similarity search.
  11. Step 3: Example construction.
  12. Step 4: ICL generation.
  13. Step 5: Initial repair until renderable.
  14. Step 6: Diagram improvement.
  15. Steps 7: Final repair until renderable.
  16. ...and 23 more sections

Figures (16)

  • Figure 1: Distribution of faithfulness hallucinations across logical organization, connectivity, and layout aesthetic scores.
  • Figure 2: (a) Hallucination-free rates across text context difficulty levels: 'H in RST' stands for $H_1$, 'H in similarity search' -- $H_2$, 'H in refinement' -- $H_6$, 'Inherited H' -- $H_{inh}$, 'Factual H' -- $H_{fact}$. (b) Faithfulness hallucination $H_{faith}$ scores per difficulty level show the number of hallucination types, where a 0 means no hallucination and a 3 means all 3 types are present in a diagram.
  • Figure 3: Example 1 based on sorva.
  • Figure 4: Example 2 based on hellas.
  • Figure 5: Example 3 based on sorva.
  • ...and 11 more figures