Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capturing Sparks of Abstraction for the ARC Challenge

Martin Andrews

TL;DR

It is demonstrated that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize.

Abstract

Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize. Both the arc-dsl-llm DSL framework (with the re-engineered solutions) and the Gemini LLM-generated data (along with the generation code) are made Open Source.

Capturing Sparks of Abstraction for the ARC Challenge

TL;DR

It is demonstrated that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize.

Abstract

Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize. Both the arc-dsl-llm DSL framework (with the re-engineered solutions) and the Gemini LLM-generated data (along with the generation code) are made Open Source.

Paper Structure

This paper contains 33 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Core Knowledge
  5. Domain Specific Language (DSL) for ARC
  6. LLM Capabilities
  7. Problem tactics
  8. Code Generation
  9. Methods
  10. Core Knowledge
  11. LLM-legible DSL
  12. Actual coded solutions
  13. Code comments
  14. Language Model targets
  15. Results
  16. ...and 18 more sections

Figures (7)

  • Figure 1: Information flow from Task 00d62c1b to Dataset
  • Figure 2: Code comparison between original, and LLM-legible versions of the same code
  • Figure 3: Line-by-line commentary generated by the LLM in the specified format
  • Figure 4: Refactoring generated by the LLM in the specified format
  • Figure 5: High-Level Tactics suggested for the sample problem
  • ...and 2 more figures