Programming Puzzles

TL;DR

Python Programming Puzzles (P3) introduces a formal, objective evaluation framework for program synthesis using verifiers $f$ and inputs $x$, and releases an open-source dataset of 397 Python puzzles spanning diverse domains from simple syntax to longstanding open problems. The paper develops enumerative AST-based solvers and autoregressive LM solvers (GPT-3 and Codex), showing bootstrapping from past solutions improves performance, with Codex solving up to about 80% of problems given enough tries. A small user study indicates puzzle solving correlates with coding experience, and that human and AI difficulty align, validating puzzles as a benchmark for algorithmic problem-solving progress. The dataset and baselines provide a platform to push advances in program synthesis and could influence code completion, automatic debugging, and the exploration of hard algorithmic problems.

Abstract

We introduce a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3). Each puzzle is defined by a short Python program $f$, and the goal is to find an input which makes $f$ return True. The puzzles are objective in that each one is specified entirely by the source code of its verifier $f$, so evaluating $f$ is all that is needed to test a candidate solution. They do not require an answer key or input/output examples, nor do they depend on natural language understanding. The dataset is comprehensive in that it spans problems of a range of difficulties and domains, ranging from trivial string manipulation problems, to classic programming puzzles (e.g., Tower of Hanoi), to interview/competitive-programming problems (e.g., dynamic programming), to longstanding open problems in algorithms and mathematics (e.g., factoring). We develop baseline enumerative program synthesis, GPT-3 and Codex solvers that are capable of solving puzzles -- even without access to any reference solutions -- by learning from their own past solutions. Codex performs best, solving up to 18% of 397 test problems with a single try and 80% of the problems with 1,000 tries per problem. In a small user study, we find a positive correlation between puzzle-solving performance and coding experience, and between the puzzle difficulty for humans and AI solvers. Therefore, further improvements on P3 could have a significant impact on many program synthesis areas.

Figures (28)

• Figure 1: Programming puzzles ranging from trivial to longstanding open algorithmic challenges in multiple domains. is solved by , a recursive program (see Figure \ref{['fig:hanoi']} on page \ref{['fig:hanoi']}) outputting 255 moves solves , and requires computational number theory algorithms.
• Figure 2: GitHub Copilot code completion examples (in gray). Left: Copilot correctly implements a seven-line function. Top right: the completion adds a space character that may or may not have been intended by the user. Middle and bottom right: errors indicating a lack of basic understanding.
• Figure 3: A Short prompt for a puzzle requesting a list of ten integers where the fourth item occurs exactly twice, with valid completion $\ldots$. Appendix \ref{['sec:gpt3_details']} has Medium/Long prompts.
• Figure 4: Increasing the number of tries allows solving new problems. Better solvers, though, solve new problems significantly faster by learning from past experience. Parametric enumerative solvers (a) initialized with the solutions of the uniform solver at $k=10^4$ accelerate the solution search. Additional self-training bootstrapping cycles (marked with B.) solve even more problems. GPT-3 (b) and Codex Davinci (c) solvers were evaluated with up to $10^4$ attempts. Having natural language descriptions (Long) provides small improvements over Medium. Adding previously found solutions to the prompt (Bootstrap) allows significant improvements for enumerative and GPT-3, and matches Long for Codex. Overall, the Codex models performed best, solving up to 127 of the examined 138 puzzles. (a), (b) are averaged across three runs and the shaded areas show the standard deviation.
• Figure 5: Number of solved puzzles by Codex-davinci (blue bars), compared to human coders with 6 minutes per puzzle (horizontal lines).

Theorems & Definitions (1)

• proof