Extraction of Research Objectives, Machine Learning Model Names, and Dataset Names from Academic Papers and Analysis of Their Interrelationships Using LLM and Network Analysis

TL;DR

The paper addresses the challenge of selecting appropriate data and machine learning methods by jointly extracting research objectives, model names, and datasets from papers and analyzing their interrelationships. It proposes an LLM-based extraction pipeline (Llama2/3) with embedding-based synonym consolidation (E5) and co-occurrence graph/network clustering to map objective–model–dataset triads. The study demonstrates practical utility in quantitative-finance literature, achieving F1 scores above 0.8 for expression extraction and revealing meaningful trends such as SP500-based stock-price datasets and ESG-related data. The approach supports future automatic recommendations of datasets and models tailored to research objectives, enhancing decision-making and knowledge discovery across domains.

Abstract

Machine learning is widely utilized across various industries. Identifying the appropriate machine learning models and datasets for specific tasks is crucial for the effective industrial application of machine learning. However, this requires expertise in both machine learning and the relevant domain, leading to a high learning cost. Therefore, research focused on extracting combinations of tasks, machine learning models, and datasets from academic papers is critically important, as it can facilitate the automatic recommendation of suitable methods. Conventional information extraction methods from academic papers have been limited to identifying machine learning models and other entities as named entities. To address this issue, this study proposes a methodology extracting tasks, machine learning methods, and dataset names from scientific papers and analyzing the relationships between these information by using LLM, embedding model, and network clustering. The proposed method's expression extraction performance, when using Llama3, achieves an F-score exceeding 0.8 across various categories, confirming its practical utility. Benchmarking results on financial domain papers have demonstrated the effectiveness of this method, providing insights into the use of the latest datasets, including those related to ESG (Environmental, Social, and Governance) data.

Figures (8)

• Figure 1: Research Overview Diagram
• Figure 2: Prompt used for data set name extraction
• Figure 3: Image of embedding
• Figure 4: Image of clustering
• Figure 5: Image of Network clustering