DistALANER: Distantly Supervised Active Learning Augmented Named Entity Recognition in the Open Source Software Ecosystem

TL;DR

This work tackles the scarcity of labeled data for domain-specific NER in the open-source software ecosystem by introducing DistALANER, a three-stage distantly supervised framework that combines dictionary construction, entity distillation with active learning, and training of multiple NER models on silver data. The approach leverages external knowledge sources and POS-based heuristics, supplemented by human validation and TagMe expansions, to build rich OSS entity dictionaries and improve recall for nine entity types. Empirical results show that DistALANER, especially under a Human-Involved setting, outperforms baseline methods and pre-LLM models, while large language models in zero-shot settings lag behind, though UniversalNER remains a competitive baseline in the LLM era. Additionally, the paper demonstrates that robust NER in OSS benefits downstream relation extraction, supporting several relation types and achieving strong encoder performance, with substantial implications for improved OSS analysis, tooling, and community insights.

Abstract

With the AI revolution in place, the trend for building automated systems to support professionals in different domains such as the open source software systems, healthcare systems, banking systems, transportation systems and many others have become increasingly prominent. A crucial requirement in the automation of support tools for such systems is the early identification of named entities, which serves as a foundation for developing specialized functionalities. However, due to the specific nature of each domain, different technical terminologies and specialized languages, expert annotation of available data becomes expensive and challenging. In light of these challenges, this paper proposes a novel named entity recognition (NER) technique specifically tailored for the open-source software systems. Our approach aims to address the scarcity of annotated software data by employing a comprehensive two-step distantly supervised annotation process. This process strategically leverages language heuristics, unique lookup tables, external knowledge sources, and an active learning approach. By harnessing these powerful techniques, we not only enhance model performance but also effectively mitigate the limitations associated with cost and the scarcity of expert annotators. It is noteworthy that our model significantly outperforms the state-of-the-art LLMs by a substantial margin. We also show the effectiveness of NER in the downstream task of relation extraction.

Figures (4)

• Figure 1: Annotation we obtain from DistALANER vs ground truth.
• Figure 2: Overview of DistALANER. Stage I annotation involves "dictionary matching" and "POS tagging". Stage II annotation then involves "dictionary expansion". After these stages, we identify four types of extractions from data. The light yellow box represents extracted "entities", while the orange box represents "error" types. The blue and dark yellow boxes represent "POS tags" and "Wikipedia mentions" respectively. We mark these exact types through links for some real bug samples.
• Figure 3: The proportion of entities recognized in Stage I and followed by the proportion extracted from Stage II.
• Figure 4: Recall rate for varying percentage of training data.