Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Matter of Representation: Towards Graph-Based Abstract Code Generation

Nyx Iskandar, Hisham Bedri, Andy Tsen

TL;DR

This work addresses graph-based abstract code generation, countering the dominance of sequential code generation by showing that LLMs can produce executable code graphs when provided with carefully designed JSON representations. The authors introduce two representations—Reference Node Representation and Output Graph Representation—and validate them on ScratchTest, a Python-implemented Scratch benchmark. Ablation studies reveal that the proposed representations significantly improve accuracy over baselines, establishing a strong baseline for graph-space code generation without fine-tuning. The findings underscore the critical role of representation design in steering LLMs toward correct, executable code graphs and lay groundwork for future representation-learning and scaling to larger graph-based programming paradigms.

Abstract

Most large language models (LLMs) today excel at generating raw, sequential code with minimal abstractions and custom structures. However, there has been little work on graph-based abstract code generation, where significant logic is encapsulated in predefined nodes and execution flow is determined by edges. This is relevant for visual programming languages, and in cases where raw source code is inaccessible to users and LLM training sets. In this work, we propose and evaluate JSON representations for graphs to enable high accuracy graph-based abstract code generation. We evaluate these representations on ScratchTest, a mini-benchmark based on our custom Python re-implementation of Scratch, which tests the LLM in code graph space. Our findings demonstrate that LLMs can indeed perform the aforementioned generation task in a single pass without relying on specialized or complex pipelines, given the correct graph representations. We also show that different representations induce significantly different accuracies, highlighting the instrumental role of representations in this generation task. All in all, this work establishes the first steps towards representation learning for graph-based abstract code generation.

A Matter of Representation: Towards Graph-Based Abstract Code Generation

TL;DR

This work addresses graph-based abstract code generation, countering the dominance of sequential code generation by showing that LLMs can produce executable code graphs when provided with carefully designed JSON representations. The authors introduce two representations—Reference Node Representation and Output Graph Representation—and validate them on ScratchTest, a Python-implemented Scratch benchmark. Ablation studies reveal that the proposed representations significantly improve accuracy over baselines, establishing a strong baseline for graph-space code generation without fine-tuning. The findings underscore the critical role of representation design in steering LLMs toward correct, executable code graphs and lay groundwork for future representation-learning and scaling to larger graph-based programming paradigms.

Abstract

Most large language models (LLMs) today excel at generating raw, sequential code with minimal abstractions and custom structures. However, there has been little work on graph-based abstract code generation, where significant logic is encapsulated in predefined nodes and execution flow is determined by edges. This is relevant for visual programming languages, and in cases where raw source code is inaccessible to users and LLM training sets. In this work, we propose and evaluate JSON representations for graphs to enable high accuracy graph-based abstract code generation. We evaluate these representations on ScratchTest, a mini-benchmark based on our custom Python re-implementation of Scratch, which tests the LLM in code graph space. Our findings demonstrate that LLMs can indeed perform the aforementioned generation task in a single pass without relying on specialized or complex pipelines, given the correct graph representations. We also show that different representations induce significantly different accuracies, highlighting the instrumental role of representations in this generation task. All in all, this work establishes the first steps towards representation learning for graph-based abstract code generation.
Paper Structure (37 sections, 2 figures, 9 tables)

This paper contains 37 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Understanding and Generation
  4. LLM Coding Agents
  5. ScratchTest
  6. Representations
  7. Reference Node Representation
  8. Output Graph Representation
  9. Evaluation
  10. Reference Node Representation
  11. Output Graph Representation
  12. Limitations
  13. Conclusion
  14. All ScratchTest Blocks
  15. LLM Configurations
  16. ...and 22 more sections

Figures (2)

  • Figure 1: Two Scratch blocks from each built-in block type in the Scratch online editor.
  • Figure 2: Correct output graph, behavior log, sprite state, and corresponding Pythonic Scratch for prompt "When the green flag is clicked, continuously move in a square pattern until the user presses the space key."