Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis

Ethan Young, Zichun Wang, Aiden Taylor, Chance Jewell, Julian Myers, Satya Sri Rajiteswari Nimmagadda, Anthony White, Aniruddha Maiti, Ananya Jana

TL;DR

It is found descriptions generated directly from images using vision-language models are often incorrect and human correction can substantially improve the quality of vision language model generated descriptions.

Abstract

Diagrams are widely used in teaching computer science courses. They are useful in subjects such as automata and formal languages, data structures, etc. These diagrams, often drawn by students during exams or assignments, vary in structure, layout, and correctness. This study examines whether current vision-language and large language models can process such diagrams and produce accurate textual and digital representations. In this study, scanned student-drawn diagrams are used as input. Then, textual descriptions are generated from these images using a vision-language model. The descriptions are checked and revised by human reviewers to make them accurate. Both the generated and the revised descriptions are then fed to a large language model to generate TikZ code. The resulting diagrams are compiled and then evaluated against the original scanned diagrams. We found descriptions generated directly from images using vision-language models are often incorrect and human correction can substantially improve the quality of vision language model generated descriptions. This research can help computer science education by paving the way for automated grading and feedback and creating more accessible instructional materials.

Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis

TL;DR

It is found descriptions generated directly from images using vision-language models are often incorrect and human correction can substantially improve the quality of vision language model generated descriptions.

Abstract

Diagrams are widely used in teaching computer science courses. They are useful in subjects such as automata and formal languages, data structures, etc. These diagrams, often drawn by students during exams or assignments, vary in structure, layout, and correctness. This study examines whether current vision-language and large language models can process such diagrams and produce accurate textual and digital representations. In this study, scanned student-drawn diagrams are used as input. Then, textual descriptions are generated from these images using a vision-language model. The descriptions are checked and revised by human reviewers to make them accurate. Both the generated and the revised descriptions are then fed to a large language model to generate TikZ code. The resulting diagrams are compiled and then evaluated against the original scanned diagrams. We found descriptions generated directly from images using vision-language models are often incorrect and human correction can substantially improve the quality of vision language model generated descriptions. This research can help computer science education by paving the way for automated grading and feedback and creating more accessible instructional materials.
Paper Structure (18 sections, 4 figures, 3 tables)

This paper contains 18 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Preparation
  4. Image scan collection
  5. Prompt Design for Description Generation
  6. Generation of Human Edited Version of the Description
  7. Experiments
  8. Semantic Distance Between Descriptions
  9. Human Evaluation of Images Generated Using Descriptions
  10. Human Evaluation of TikZ Compiled Diagrams
  11. Results
  12. Semantic Similarity Between Vision-Language Generated Text and Human-Edited Description
  13. TikZ Compilation Rate
  14. Human Evaluation of Reconstruction Quality Using Diagrams Directly Generated from Text Descriptions
  15. Human Evaluation of Reconstruction Quality Using Diagrams Generated Via TikZ Code
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Sample of student-drawn diagram related to Turing machine execution trace
  • Figure 2: Hand-drawn diagram (left), diagram from original description (middle) and diagram from edited description (right)
  • Figure 3: Hand-drawn diagram (left), diagram from original description (middle) and diagram from edited description(right)
  • Figure 4: Hand-drawn diagram (left), TikZ diagram from original description (middle) and TikZ diagram from edited description (right)