Publication Trend Analysis and Synthesis via Large Language Model: A Case Study of Engineering in PNAS

TL;DR

This work tackles the challenge of mapping the evolving landscape of science beyond static journal classifications by applying a two-stage, LLM-driven framework to 1,519 PNAS Engineering articles (2005–2024). It first derives 16 primary abstract topics via iterative clustering and GPT-4o mini labeling, then extends to full-text segments with multi-label classifications, linking abstract and full-text themes through a bipartite graph. Validations with BoW and c-TF-IDF demonstrate semantic coherence and linguistic structure, while comparisons to the journal’s dual classifications show substantial alignment and reveal cross-disciplinary connections that static keywords miss. The approach yields a dynamic, interpretable map of thematic structure and inter-topic flows, offering a scalable tool for trend detection that can be reapplied to other journals and evolving scientific domains.

Abstract

Scientific literature is increasingly siloed by complex language, static disciplinary structures, and potentially sparse keyword systems, making it cumbersome to capture the dynamic nature of modern science. This study addresses these challenges by introducing an adaptable large language model (LLM)-driven framework to quantify thematic trends and map the evolving landscape of scientific knowledge. The approach is demonstrated over a 20-year collection of more than 1,500 engineering articles published by the Proceedings of the National Academy of Sciences (PNAS), marked for their breadth and depth of research focus. A two-stage classification pipeline first establishes a primary thematic category for each article based on its abstract. The subsequent phase performs a full-text analysis to assign secondary classifications, revealing latent, cross-topic connections across the corpus. Traditional natural language processing (NLP) methods, such as Bag-of-Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF), confirm the resulting topical structure and also suggest that standalone word-frequency analyses may be insufficient for mapping fields with high diversity. Finally, a disjoint graph representation between the primary and secondary classifications reveals implicit connections between themes that may be less apparent when analyzing abstracts or keywords alone. The findings show that the approach independently recovers much of the journal's editorially embedded structure without prior knowledge of its existing dual-classification schema (e.g., biological studies also classified as engineering). This framework offers a powerful tool for detecting potential thematic trends and providing a high-level overview of scientific progress.

Figures (10)

• Figure 1: Overview of methodology.
• Figure 2: Cumulative (a) and overlapping (b) distributions of top LLM-derived topics and classifications for engineering abstracts. Best viewed in color.
• Figure 3: Word frequency ($f$) versus rank ($r$) for all abstracts on linear scale (a) and log-log scale with normalized frequency (b) conforming to Zipf’s law. The highlighted area under each curve represents the Pareto region, where 80% of the most frequent words occur in the first 20% of rankings.
• Figure 4: Progressive c-TF-IDF word clouds of LLM-derived topic abstract classifications. The x-axis shows four bucketed time periods spanning five years each. The y-axis shows PNAS author-selected keywords over the entire engineering subcategory, the top six research topics from abstract classification, and the final “Other” category. The dots are a placeholder for ten additional topics between “Electronic Innovations” and “Other,” excluded due to spatial limitations.
• Figure 5: Distribution of LLM-derived topic classifications for PNAS engineering abstracts and full-text segments. The outer ring (abstract classification) shows the distribution of 1,519 across 16 topics derived from an iterative clustering and reclassification procedure. The inner ring (full-text classification) depicts the distribution of over 18,000 full-text segments from the articles classified into the same set of 16 topics. The two distributions differ due to the finer-grained and multi-label nature of the full-text classification process.