Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Good things come in three: Generating SO Post Titles with Pre-Trained Models, Self Improvement and Post Ranking

Duc Anh Le, Anh M. T. Bui, Phuong T. Nguyen, Davide Di Ruscio

TL;DR

FILLER has the potential to be used in practice to support developers in generating suitable post titles in Stack Overflow, and significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo.

Abstract

Stack Overflow is a prominent Q and A forum, supporting developers in seeking suitable resources on programming-related matters. Having high-quality question titles is an effective means to attract developers' attention. Unfortunately, this is often underestimated, leaving room for improvement. Research has been conducted, predominantly leveraging pre-trained models to generate titles from code snippets and problem descriptions. Yet, getting high-quality titles is still a challenging task, attributed to both the quality of the input data (e.g., containing noise and ambiguity) and inherent constraints in sequence generation models. In this paper, we present FILLER as a solution to generating Stack Overflow post titles using a fine-tuned language model with self-improvement and post ranking. Our study focuses on enhancing pre-trained language models for generating titles for Stack Overflow posts, employing a training and subsequent fine-tuning paradigm for these models. To this end, we integrate the model's predictions into the training process, enabling it to learn from its errors, thereby lessening the effects of exposure bias. Moreover, we apply a post-ranking method to produce a variety of sample candidates, subsequently selecting the most suitable one. To evaluate FILLER, we perform experiments using benchmark datasets, and the empirical findings indicate that our model provides high-quality recommendations. Moreover, it significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo. A user study also shows that FILLER provides more relevant titles, with respect to SOTitle and GPT3.5-turbo.

Good things come in three: Generating SO Post Titles with Pre-Trained Models, Self Improvement and Post Ranking

TL;DR

FILLER has the potential to be used in practice to support developers in generating suitable post titles in Stack Overflow, and significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo.

Abstract

Stack Overflow is a prominent Q and A forum, supporting developers in seeking suitable resources on programming-related matters. Having high-quality question titles is an effective means to attract developers' attention. Unfortunately, this is often underestimated, leaving room for improvement. Research has been conducted, predominantly leveraging pre-trained models to generate titles from code snippets and problem descriptions. Yet, getting high-quality titles is still a challenging task, attributed to both the quality of the input data (e.g., containing noise and ambiguity) and inherent constraints in sequence generation models. In this paper, we present FILLER as a solution to generating Stack Overflow post titles using a fine-tuned language model with self-improvement and post ranking. Our study focuses on enhancing pre-trained language models for generating titles for Stack Overflow posts, employing a training and subsequent fine-tuning paradigm for these models. To this end, we integrate the model's predictions into the training process, enabling it to learn from its errors, thereby lessening the effects of exposure bias. Moreover, we apply a post-ranking method to produce a variety of sample candidates, subsequently selecting the most suitable one. To evaluate FILLER, we perform experiments using benchmark datasets, and the empirical findings indicate that our model provides high-quality recommendations. Moreover, it significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo. A user study also shows that FILLER provides more relevant titles, with respect to SOTitle and GPT3.5-turbo.
Paper Structure (20 sections, 9 equations, 5 figures, 7 tables)

This paper contains 20 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. BACKGROUND AND MOTIVATION
  3. Related Work
  4. Challenges
  5. PROPOSED SOLUTION
  6. Fine-tuning PTM with multi-modal inputs
  7. Self Improvement
  8. Post Ranking
  9. EMPIRICAL EVALUATION
  10. Research Questions
  11. Dataset
  12. Evaluation Metrics
  13. Implementation Details
  14. EMPIRICAL RESULTS
  15. RQ$_1$: How do Pre-trained Models influence the performance of Stack Overflow post title generation?
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Titles generated by CodeT5: The firstly ranked title is not the most relevant one.
  • Figure 2: The architecture of FILLER.
  • Figure 3: Pseudo code of the Self--Improvement Process
  • Figure 4: The title generated by FILLER compared to those from the baselines for a question post example.
  • Figure 5: Evaluation results from the user study.