Once Upon an Input: Reasoning via Per-Instance Program Synthesis

TL;DR

This work addresses the difficulty of multi-step reasoning in large language models by introducing Per-Instance Program Synthesis (PIPS), which selectively synthesizes and refines executable programs at the instance level. A learned confidence switch decides, per instance, whether to use direct inference via chain-of-thought or to generate a structured program, thereby avoiding unnecessary code generation on non-algorithmic tasks. PIPS iteratively refines programs through structural feedback and explicit instance-specific symbolic extraction, reducing undesirable code and improving solution correctness across 30 frontier datasets and multiple LLMs. The approach yields up to 8.6% absolute harmonic mean accuracy gains over Program of Thought and 9.4% over Chain of Thought, while maintaining or improving performance on non-algorithmic tasks and reducing brittle handling of unstructured inputs. These results suggest that instance-level synthesis, guided by robust feedback and symbol extraction, is a promising direction for scalable, reliable reasoning in AI systems.

Abstract

Large language models (LLMs) excel at zero-shot inference but continue to struggle with complex, multi-step reasoning. Recent methods that augment LLMs with intermediate reasoning steps such as Chain of Thought (CoT) and Program of Thought (PoT) improve performance but often produce undesirable solutions, especially in algorithmic domains. We introduce Per-Instance Program Synthesis (PIPS), a method that generates and refines programs at the instance-level using structural feedback without relying on task-specific guidance or explicit test cases. To further improve performance, PIPS incorporates a confidence metric that dynamically chooses between direct inference and program synthesis on a per-instance basis. Experiments across three frontier LLMs and 30 benchmarks including all tasks of Big Bench Extra Hard (BBEH), visual question answering tasks, relational reasoning tasks, and mathematical reasoning tasks show that PIPS improves the absolute harmonic mean accuracy by up to 8.6% and 9.4% compared to PoT and CoT respectively, and reduces undesirable program generations by 65.1% on the algorithmic tasks compared to PoT with Gemini-2.0-Flash.

Figures (11)

• Figure 1: Overview of Per-Instance Program Synthesis ( PIPS ). PIPS addresses the open-domain nature of reasoning problems by selecting between synthesis and CoT at the instance-level, avoiding unnecessarily generating programs for non-algorithmic problems. For algorithmic problems, PIPS addresses the lack of task specifications by iteratively synthesizing programs using feedback based on structural checks. PIPS handles unstructured input via instance-specific symbolic extraction (step 1) before program synthesis (step 2). \ref{['fig:synth-loop']} shows an example where an undesirable program is rejected before producing an acceptable one which gives the correct answer upon execution (step 3).
• Figure 2: Example illustrating two iterations of the synthesis loop in PIPS .
• Figure 3: Failures in existing approaches to per-instance program synthesis with Gemini-2.0-Flash.
• Figure 4: Two programs generated with PoT illustrating program synthesis failures. Part (a) shows a trivial program where two variables are initialized to zero, but then several steps of reasoning are performed in comments, leading to their values being hard-coded rather than computed with code. Part (b) shows an input-free program to process the input image itself which would be better done using the LLM's perceptual inference. Both programs result in the wrong answer. The corresponding programs produced by PIPS yield the correct answer in both cases and are shown in \ref{['app:our-programs']}.
• Figure 5: Harmonic mean accuracy over all 30 datasets (left), on the 17 majority algorithmic tasks (middle), and on the 10 majority non-algorithmic (right) for PIPS and baselines using three state-of-the-art models. The breakdown per task per model is shown in \ref{['tab:gemini_20_flash_details']}, \ref{['tab:o4_mini_2025_04_16_details']}, and \ref{['tab:gpt_41_mini_2025_04_14_details']}.