SELU: A Software Engineering Language Understanding Benchmark

TL;DR

SELU introduces the first comprehensive benchmark for evaluating LLMs on SE textual artifacts NL tasks across the SE life cycle, combining 22 tasks with a Bayesian pairwise comparison framework. It demonstrates that fine-tuned open-source LLMs generally outperform baselines and proprietary prompting, while domain-adaptation via code-focused pre-training yields limited benefits. The work provides practical guidance favoring mid-sized open-source models for SE textual NLU and identifies future directions toward broader SE-generation tasks, industrial data, and rigorous reproducibility. Overall, SELU lays groundwork for robust, cross-task evaluation of LLMs in diverse SE text-centric activities and informs deployment choices in practice.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation. However, their effectiveness on non-code Software Engineering (SE) tasks remains underexplored. We present 'Software Engineering Language Understanding' (SELU), the first comprehensive benchmark for evaluating LLMs on 22 SE textual artifacts NLU tasks, spanning from identifying whether a requirement is functional or non-functional to estimating the effort required to implement a development task. SELU covers classification, regression, Named Entity Recognition (NER), and Masked Language Modeling (MLM) tasks, with data drawn from diverse sources such as issue tracking systems and developer forums. We fine-tune 22 open-source LLMs, both generalist and domain-adapted; and prompt two proprietary alternatives using zero-shot a 3-shot prompting strategies. Performance is measured using metrics such as F1-macro, SMAPE, F1-micro, and accuracy, and compared via the Bayesian signed-rank test. Our results show that fine-tuned models across various sizes and architectures perform best, exhibiting high mean performance and low across-task variance. Furthermore, domain adaptation via code-focused pre-training does not yield significant improvements and might even be counterproductive for developer communication tasks.

Figures (6)

• Figure 1: Distribution of published works per SE life cycle activity. Adapted from Hou et al. llm4se2.
• Figure 2: Distribution of tasks per SE life cycle activity in SELU.
• Figure 3: LLM heads for different task types.
• Figure 4: Base prompt template generally applied across tasks. New lines partially omitted to reduce space.
• Figure 5: Posterior probability matrices of (a) 'A outperforming B,' (b) 'B outperforming A,' and (c) 'A and B being practically equivalent.' (d) Represents inconclusive result (i.e., none of the probabilities is higher than $0.95$).