Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jigsaw: Large Language Models meet Program Synthesis

Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, Rahul Sharma

TL;DR

This work tackles the gap that large language models can generate code but cannot guarantee correctness. It introduces Jigsaw, a multi-modal code-synthesis framework that combines natural-language intent and I/O examples with pre-processing and post-processing steps to enforce syntactic and semantic correctness for Pandas API tasks, and it learns from user feedback to improve over time. Empirical results on PandasEval1 and PandasEval2 show large accuracy gains over vanilla PTLMs and AutoPandas, driven by context-aware prompting, variable/argument transformations, and AST-to-AST repairs learned via PROSE Refazer. The findings suggest that integrating program-analysis-based post-processing with multi-modal inputs substantially enhances developer productivity and can generalize beyond Pandas to other libraries and APIs, while highlighting remaining challenges in scalability, security, and robust, language-agnostic extension.

Abstract

Large pre-trained language models such as GPT-3, Codex, and Google's language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, jigsaw has an important role to play in improving the accuracy of the systems.

Jigsaw: Large Language Models meet Program Synthesis

TL;DR

This work tackles the gap that large language models can generate code but cannot guarantee correctness. It introduces Jigsaw, a multi-modal code-synthesis framework that combines natural-language intent and I/O examples with pre-processing and post-processing steps to enforce syntactic and semantic correctness for Pandas API tasks, and it learns from user feedback to improve over time. Empirical results on PandasEval1 and PandasEval2 show large accuracy gains over vanilla PTLMs and AutoPandas, driven by context-aware prompting, variable/argument transformations, and AST-to-AST repairs learned via PROSE Refazer. The findings suggest that integrating program-analysis-based post-processing with multi-modal inputs substantially enhances developer productivity and can generalize beyond Pandas to other libraries and APIs, while highlighting remaining challenges in scalability, security, and robust, language-agnostic extension.

Abstract

Large pre-trained language models such as GPT-3, Codex, and Google's language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, jigsaw has an important role to play in improving the accuracy of the systems.
Paper Structure (23 sections, 5 figures, 8 tables, 1 algorithm)

This paper contains 23 sections, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Jigsaw Overview
  3. $\mathsf{Jigsaw\xspace}$ design principles
  4. $\mathsf{PTLM}$s as black-box
  5. Learning to fix recurring failure modes of $\mathsf{PTLM}$s
  6. Jigsaw Architecture
  7. Pre-trained Language Models
  8. Pre-processing
  9. Post-processing
  10. Learning from user feedback
  11. Datasets
  12. PandasEval dataset $\mathsf{PandasEval1\xspace}$
  13. Hackathon dataset $\mathsf{PandasEval2\xspace}$
  14. Experiments
  15. Offline evaluation
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Multi-modal problem specification in $\mathsf{Jigsaw\xspace}$
  • Figure 2: Architecture of Jigsaw
  • Figure 3: English to French translation using $\mathsf{GPT\text{-}3}$
  • Figure 4: Illustration of Pre-processing step of $\mathsf{Jigsaw\xspace}$
  • Figure 5: Example task, part of the dataset $\mathsf{PandasEval2\xspace}$, as presented to the user during the Hackathon session.