Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling

TL;DR

This work tackles the challenge of classifying interdisciplinary scientific abstracts amid massive publication volume by introducing a SciBERT-CNN pipeline that ingests multi-segment inputs (abstract, body, title, and topic-model-derived keywords) and uses class-weighted loss to address long-tailed label distributions. The approach, tested on the 40k-article Elsevier OA CC-BY corpus, consolidates 27 subject areas into 18 labels and achieves superior multi-label F1 and high per-label AUC (0.93–0.99) compared to a BERT baseline. Key contributions include the multi-segment input strategy, topic-model-derived keywords, a CNN for local feature extraction, and a principled class-weighting scheme $w(l) = 1 / freq(l)$ with normalization $w(l) = w(l)/\sum_{l \in L} w(l)$ and sample weights $w_s = \sum_{l \in l_s} w(l)$. The results demonstrate improved classification performance and potential for more efficient literature reviews, while acknowledging limitations of domain coverage and label granularity that motivate future work in data augmentation and domain-specific keyword integration.

Abstract

Researchers must stay current in their fields by regularly reviewing academic literature, a task complicated by the daily publication of thousands of papers. Traditional multi-label text classification methods often ignore semantic relationships and fail to address the inherent class imbalances. This paper introduces a novel approach using the SciBERT model and CNNs to systematically categorize academic abstracts from the Elsevier OA CC-BY corpus. We use a multi-segment input strategy that processes abstracts, body text, titles, and keywords obtained via BERT topic modeling through SciBERT. Here, the [CLS] token embeddings capture the contextual representation of each segment, concatenated and processed through a CNN. The CNN uses convolution and pooling to enhance feature extraction and reduce dimensionality, optimizing the data for classification. Additionally, we incorporate class weights based on label frequency to address the class imbalance, significantly improving the classification F1 score and enhancing text classification systems and literature review efficiency.

Figures (6)

• Figure 1: The 27 Classes: Subject Area Imbalance
• Figure 2: Resulting 18 Class Subject Areas
• Figure 3: SciBERT-CNN Architecture
• Figure 4: Misplaced labels
• Figure 5: Receiver Operating Characteristics of each label