Less is More: Summary of Long Instructions is Better for Program Synthesis

TL;DR

This work shows that long, detail-rich problem descriptions can hinder program-synthesis models and that concise, high-quality summaries improve Codex performance on code-generation tasks. By constructing a meta-dataset of human and synthetic summaries from APPS and CodeContests, the authors demonstrate substantial gains, particularly for introductory and interview-like problems, and provide analyses of biases and linguistic features that influence success. The main contributions include a large-scale, multi-source dataset of summarized prompts, an empirical evaluation across multiple prompting regimes, and design principles for instruction-style prompts in program synthesis. The findings highlight the practical significance of prompt engineering in reducing noise and improving reliability of machine-generated code, with future work focusing on decomposing prompts and leveraging diverse summarizers to further close the gap in more challenging tasks.

Abstract

Despite the success of large pre-trained language models (LMs) such as Codex, they show below-par performance on the larger and more complicated programming related questions. We show that LMs benefit from the summarized version of complicated questions. Our findings show that superfluous information often present in problem description such as human characters, background stories, and names (which are included to help humans in understanding a task) does not help models in understanding a task. To this extent, we create a meta-dataset from the frequently used APPS dataset and the newly created CodeContests dataset for the program synthesis task. Our meta-dataset consists of human and synthesized summaries of the long and complicated programming questions. Experimental results on Codex show that our proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on average in terms of strict accuracy. Our analysis shows that summaries significantly improve performance for introductory (9.86%) and interview (11.48%) programming questions. However, it shows improvement by a small margin (~ 2%) for competitive programming questions, implying scope for future research in this direction.

Figures (10)

• Figure 1: (Top plot) Mean frequency of POS for problems where programs where the generated by both the original and summarized prompt pass all test cases, and (Bottom plot) mean frequency of POS for problems where the summary passes all test cases and the original did not. The blue bar represents the mean of the entire dataset. analyzed only the top 11 most occurring POS. The plot shows that higher number of nouns degrade model performance.
• Figure 2: (Top plot) Mean frequency of the entity labels for problems where program generated by the original and summarized prompt pass all test cases, and (Bottom plot) mean frequency of entity labels for problems where the summary passes all test cases and the original did not. We analyzed only the top 5 most occurring entities among all entities we found.
• Figure 3: On the far left is the code generated by the original prompt. The middle is the code generated by the expert summary. The right is the code generated by the basic summary.
• Figure 4: The left is the code generated using the original prompt. The right is the code generated when using the StudioAI21 generated summary.
• Figure 5: The left is the code generated using the original prompt. The right is the code generated when using the StudioAI21 generated summary.