Measuring Emergent Capabilities of LLMs for Software Engineering: How Far Are We?

TL;DR

This paper proposes a model-agnostic pipeline for evaluating the emergence of capabilities in the context of SE, and presents a case study instantiating the pipeline to analyze the emergence of capabilities in CodeGen1-multi across four scales ranging from 350M to 16.1B parameters.

Abstract

The adoption of Large Language Models (LLMs) across multiple contexts has sparked interest in understanding how scaling model size might lead to behavioral changes, as LLMs can exhibit behaviors not observed in their smaller counterparts. Understanding these emergent capabilities is essential for advancing LLM development and improving their interpretability across diverse tasks. However, whether LLMs exhibit true emergence in the context of Software Engineering remains an unexplored topic, as most research has focused on NLP tasks. In this paper, we investigate the emergence of capabilities in the context of SE. We propose a model-agnostic pipeline for evaluating this phenomenon across three SE tasks: bug fixing, code translation, and commit message generation. More precisely, for each task, we present a case study instantiating our pipeline to analyze the emergence of capabilities in CodeGen1-multi across four scales ranging from 350M to 16.1B parameters. Our findings do not not provide evidence to support the idea of emergent capabilities resulting from scaling the model size in the selected set of tasks. We hope our results can pave the way to a more nuanced understanding of emergent capabilities of LLMs within the SE domain, guiding future research to focus on task-specific evaluations and the identification of alternative factors contributing to this phenomenon. Our work underscores the importance of task diversity in examining model behaviors and highlights potential limitations in transferring prior understandings of and approaches to emergence from NLP to Software Engineering.

Figures (11)

• Figure 1: The EmergentEval pipeline instantiated with particular test cases, four model scales, and a visualized linear regression evaluation, which produces a regression value $p$.
• Figure 2: Prompting strategy used on CodeGen1-multi for the code repair tasks.
• Figure 3: Prompting strategy used on CodeGen1-multi for the code translation tasks.
• Figure 4: Prompting strategy used on CodeGen1-multi for the commit message generation tasks. When designing this prompt, coaxing the model to output English description required finesse. In this case, during testing, we found the most consistent approach to obtaining English descriptions was to embed the diff output in a labeled, multi-line code comment, followed by two single-line comments which prompt the model with the context of English language descriptions contained within code comments.
• Figure 5: Bootstrapping with $S=50$ and $N=500$ over the CodeXGLUE Bugs2fix task, assessed by metrics BLEU and CodeBLEU, over the results of prompting CodeGen1-multi. (The Exact Match metric is omitted, as it is 0 across all data points).