Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replication in Requirements Engineering: the NLP for RE Case

Sallam Abualhaija, F. BaŞAk Aydemir, Fabiano Dalpiaz, Davide Dell'Anna, Alessio Ferrari, Xavier Franch, Davide Fucci

TL;DR

The paper tackles the limited replication support in NLP4RE by introducing the ID-Card, a structured artifact that records replication-relevant information. It grounds the ID-Card in two hands-on replication cases—anaphoric ambiguity detection and FR-NFR classification—through a design-science process and focus groups. The study identifies 16 replication challenges across dataset annotation and tool reconstruction, and demonstrates how the ID-Card can complement primary papers to improve reproducibility. The work aims to raise awareness of replication in NLP4RE and provides a practical tool to facilitate replication, education, and future artifact evaluation.

Abstract

[Context]} Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks' inherent hairiness, and, in turn, the heterogeneous reporting structure. [Solution] To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. [Results] In this paper: (i) we report on hands-on experiences of replication, (ii) we review the state-of-the-art and extract replication-relevant information, (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction, and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. [Contribution] This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication, but can also be used in other contexts, e.g., for educational purposes.

Replication in Requirements Engineering: the NLP for RE Case

TL;DR

The paper tackles the limited replication support in NLP4RE by introducing the ID-Card, a structured artifact that records replication-relevant information. It grounds the ID-Card in two hands-on replication cases—anaphoric ambiguity detection and FR-NFR classification—through a design-science process and focus groups. The study identifies 16 replication challenges across dataset annotation and tool reconstruction, and demonstrates how the ID-Card can complement primary papers to improve reproducibility. The work aims to raise awareness of replication in NLP4RE and provides a practical tool to facilitate replication, education, and future artifact evaluation.

Abstract

[Context]} Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks' inherent hairiness, and, in turn, the heterogeneous reporting structure. [Solution] To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. [Results] In this paper: (i) we report on hands-on experiences of replication, (ii) we review the state-of-the-art and extract replication-relevant information, (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction, and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. [Contribution] This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication, but can also be used in other contexts, e.g., for educational purposes.
Paper Structure (45 sections, 4 figures, 5 tables)

This paper contains 45 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research Questions and Method
  4. Reconstruction Cases of NLP4RE Solutions
  5. Anaphoric Ambiguity Detection
  6. Functional and Non-functional Requirements Classification
  7. Replication Challenges of NLP4RE Studies
  8. Challenges of Dataset Annotation and Re-annotation (RQ1)
  9. Theoretical Foundation Challenges
  10. Ann1. Some RE-specific categorization tasks lack solid theories that can guide the annotation process.
  11. Ann2. Besides annotation experience and theoretical knowledge, the lack of domain knowledge can limit the accuracy of the annotations.
  12. Annotation Process Challenges
  13. Ann3. The annotation activity is time consuming due to factors such as language barriers, different individuals' background, and fatigue.
  14. Ann4. The annotation protocol can evolve and thereby necessitate the re-annotation of the data which might, again, cause additional time and effort.
  15. Ann5. Theoretical and practical training resources and opportunities are limited and not adequate for training novice annotators who are often trained during the annotation task by more experienced annotators.
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Overview of the terminology used in this paper.
  • Figure 2: Overview of the research method of this paper.
  • Figure 3: Results of TAM assessment for the NLP4RE ID-card.
  • Figure 4: Assessment of the NLP4RE ID Card appropriateness and level of details.