ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges

TL;DR

ScratchEval presents a Scratch-based visual programming benchmark to jointly evaluate visual understanding and programming logic in large multimodal models. By combining image data with embedded Scratch script logic across 305 bilingual MCQs, the study reveals that current LMMs struggle with authentic visual code reasoning, though prompting strategies such as CoT can yield meaningful performance gains. The results identify clear weaknesses in multi-step reasoning and visual encoding, motivating future work on integrated visual-language training and robust reasoning. The dataset, derived from expert-curated sources and released under Apache 2.0, provides a practical platform for advancing visual programming capabilities in AI systems and for cross-linguistic robustness research.

Abstract

Recent advancements in large multimodal models (LMMs) have showcased impressive code generation capabilities, primarily evaluated through image-to-code benchmarks. However, these benchmarks are limited to specific visual programming scenarios where the logic reasoning and the multimodal understanding capacities are split apart. To fill this gap, we propose ScratchEval, a novel benchmark designed to evaluate the visual programming reasoning ability of LMMs. ScratchEval is based on Scratch, a block-based visual programming language widely used in children's programming education. By integrating visual elements and embedded programming logic, ScratchEval requires the model to process both visual information and code structure, thereby comprehensively evaluating its programming intent understanding ability. Our evaluation approach goes beyond the traditional image-to-code mapping and focuses on unified logical thinking and problem-solving abilities, providing a more comprehensive and challenging framework for evaluating the visual programming ability of LMMs. ScratchEval not only fills the gap in existing evaluation methods, but also provides new insights for the future development of LMMs in the field of visual programming. Our benchmark can be accessed at https://github.com/HKBUNLP/ScratchEval .

Figures (9)

• Figure 1: The illustration of the evaluation process for ScratchEval.
• Figure 2: Models's performance under different prompting strategies.
• Figure 3: A Gemini-1.5-Pro mistake case. The error areas are marked in red.
• Figure 4: Examples used in the Case study. The error areas are marked in red.
• Figure 5: Performance under different prompting strategies.