Constraint-Driven Small Language Models Based on Agent and OpenAlex Knowledge Graph: Mining Conceptual Pathways and Discovering Innovation Points in Academic Papers
Ziye Xia, Sergei S. Ospichev
TL;DR
This paper tackles information overload in academia by grounding concept-path analysis in the OpenAlex knowledge graph and guiding small language models through a four-stage, constraint-driven pipeline. By combining structured prompts, KG constraints, and human-in-the-loop validation, the approach dramatically reduces LLM hallucinations and achieves high end-to-end precision in extracting domain concepts and identifying innovation points. Key findings show that novelty often arises from rare structural combinations of mainstream concepts rather than new terms, with the Stage 3 KG constraint integration boosting precision to $95.19\%$ and F1 to $82.26\%$, and Stage 4 concept ownership judgment reaching $98.65\%$ F1. The work demonstrates a practical, reproducible framework for scholarly analysis that blends small-model fine-tuning, knowledge constraints, and interactive validation to produce interpretable concept paths and robust novelty signals.
Abstract
In recent years, the rapid increase in academic publications across various fields has posed severe challenges for academic paper analysis: scientists struggle to timely and comprehensively track the latest research findings and methodologies. Key concept extraction has proven to be an effective analytical paradigm, and its automation has been achieved with the widespread application of language models in industrial and scientific domains. However, existing paper databases are mostly limited to similarity matching and basic classification of key concepts, failing to deeply explore the relational networks between concepts. This paper is based on the OpenAlex opensource knowledge graph. By analyzing nearly 8,000 open-source paper data from Novosibirsk State University, we discovered a strong correlation between the distribution patterns of paper key concept paths and both innovation points and rare paths. We propose a prompt engineering-based key concept path analysis method. This method leverages small language models to achieve precise key concept extraction and innovation point identification, and constructs an agent based on a knowledge graph constraint mechanism to enhance analysis accuracy. Through fine-tuning of the Qwen and DeepSeek models, we achieved significant improvements in accuracy, with the models publicly available on the Hugging Face platform.