Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models

TL;DR

This paper tackles the high computational cost of large code intelligence PLMs by evaluating prompt learning and proposing GenAP, a gradient-free Genetic Algorithm-based method to automatically design discrete prompts. It shows that prompt effectiveness hinges on prompt design, with manual prompts being labor-intensive and NLP auto-prompt methods often inapplicable to code tasks. GenAP automatically initializes, encodes, and evolves prompts (including verbalizers and template order) with variable-length encoding, achieving superior performance across defect prediction, code summarization, and code translation while incurring only the inference-cost overhead. The work demonstrates GenAP’s scalability to CodeT5+-level models and establishes design patterns—especially the primacy of verbalizers and prompt order—while highlighting limitations and threats to validity, and it positions GenAP as a practical, interpretable, and efficient alternative to fine-tuning and gradient-based prompt tuning for code intelligence."

Abstract

As Pre-trained Language Models (PLMs), a popular approach for code intelligence, continue to grow in size, the computational cost of their usage has become prohibitively expensive. Prompt learning, a recent development in the field of natural language processing, emerges as a potential solution to address this challenge. In this paper, we investigate the effectiveness of prompt learning in code intelligence tasks. We unveil its reliance on manually designed prompts, which often require significant human effort and expertise. Moreover, we discover existing automatic prompt design methods are very limited to code intelligence tasks due to factors including gradient dependence, high computational demands, and limited applicability. To effectively address both issues, we propose Genetic Auto Prompt (GenAP), which utilizes an elaborate genetic algorithm to automatically design prompts. With GenAP, non-experts can effortlessly generate superior prompts compared to meticulously manual-designed ones. GenAP operates without the need for gradients or additional computational costs, rendering it gradient-free and cost-effective. Moreover, GenAP supports both understanding and generation types of code intelligence tasks, exhibiting great applicability. We conduct GenAP on three popular code intelligence PLMs with three canonical code intelligence tasks including defect prediction, code summarization, and code translation. The results suggest that GenAP can effectively automate the process of designing prompts. Specifically, GenAP outperforms all other methods across all three tasks (e.g., improving accuracy by an average of 2.13% for defect prediction). To the best of our knowledge, GenAP is the first work to automatically design prompts for code intelligence PLMs.

Figures (7)

• Figure 1: Illustration on the pipeline of Inference without Prompt (a) and Prompt Learning (b), take defect prediction as an example.
• Figure 2: Illustration on the paradigms of Discrete Prompt Learning (a), Soft Prompt Learning (b), and Discrete Prompt Auto-design (c).
• Figure 3: Illustration on the pipeline of GenAP, take defect prediction as an example
• Figure 4: Ablation results with different operator settings. Note that axis 0 denotes the result of GAP w/o manual prompt initialization.
• Figure 5: Memory Usage with prompt design methods when applying CodeBERT to defect prediction.