Studying and Recommending Information Highlighting in Stack Overflow Answers

TL;DR

The paper addresses how information is highlighted in Stack Overflow answers and whether such highlighting can be automatically recommended. It combines a large-scale empirical study of 31,169,429 answers with CNN- and BERT-based NER-inspired models to identify content deserving formatting across Code, Bold, Italic, Heading, and Delete types. Key findings show highlighting is prevalent (47.6%), Code is the most common focus, and CNN-based models, especially for Code, achieve higher precision (up to 0.72) and F1 (0.69) than BERT, though recall remains challenging for non-code formats. The work demonstrates practical usefulness through post-level evaluations and outlines avenues for improving downstream tasks like answer summarization and API documentation enrichment, while providing replication resources.

Abstract

Context: Navigating the knowledge of Stack Overflow (SO) remains challenging. To make the posts vivid to users, SO allows users to write and edit posts with Markdown or HTML so that users can leverage various formatting styles (e.g., bold, italic, and code) to highlight the important information. Nonetheless, there have been limited studies on the highlighted information. Objective: We carried out the first large-scale exploratory study on the information highlighted in SO answers in our recent study. To extend our previous study, we develop approaches to automatically recommend highlighted content with formatting styles using neural network architectures initially designed for the Named Entity Recognition task. Method: In this paper, we studied 31,169,429 answers of Stack Overflow. For training recommendation models, we choose CNN-based and BERT-based models for each type of formatting (i.e., Bold, Italic, Code, and Heading) using the information highlighting dataset we collected from SO answers. Results: Our models achieve a precision ranging from 0.50 to 0.72 for different formatting types. It is easier to build a model to recommend Code than other types. Models for text formatting types (i.e., Heading, Bold, and Italic) suffer low recall. Our analysis of failure cases indicates that the majority of the failure cases are due to missing identification. One explanation is that the models are easy to learn the frequent highlighted words while struggling to learn less frequent words (i.g., long-tail knowledge). Conclusion: Our findings suggest that it is possible to develop recommendation models for highlighting information for answers with different formatting styles on Stack Overflow.

Figures (7)

• Figure 1: An example of SO answers in which the code-related content is highlighted.
• Figure 2: An example of revising highlight to improve the presentation of the post.
• Figure 3: Framework of our study.
• Figure 4: Layers of the CNN model
• Figure 5: Layers of the NER BERT model