Discovering emergent connections in quantum physics research via dynamic word embeddings

TL;DR

This work introduces a novel approach based on dynamic word embeddings for concept combination prediction that captures implicit relationships between concepts, can be learned in a fully unsupervised manner, and encodes a broader spectrum of information.

Abstract

As the field of quantum physics evolves, researchers naturally form subgroups focusing on specialized problems. While this encourages in-depth exploration, it can limit the exchange of ideas across structurally similar problems in different subfields. To encourage cross-talk among these different specialized areas, data-driven approaches using machine learning have recently shown promise to uncover meaningful connections between research concepts, promoting cross-disciplinary innovation. Current state-of-the-art approaches represent concepts using knowledge graphs and frame the task as a link prediction problem, where connections between concepts are explicitly modeled. In this work, we introduce a novel approach based on dynamic word embeddings for concept combination prediction. Unlike knowledge graphs, our method captures implicit relationships between concepts, can be learned in a fully unsupervised manner, and encodes a broader spectrum of information. We demonstrate that this representation enables accurate predictions about the co-occurrence of concepts within research abstracts over time. To validate the effectiveness of our approach, we provide a comprehensive benchmark against existing methods and offer insights into the interpretability of these embeddings, particularly in the context of quantum physics research. Our findings suggest that this representation offers a more flexible and informative way of modeling conceptual relationships in scientific literature.

Figures (6)

• Figure 1: Overview: (a) We analyze a dataset of $66,839$ papers with the quant-ph identifier on arXiv, spanning from $1994$ to $2023$. From these papers, we extract $10,235$ quantum physics-related concepts using RAKE and other NLP tools. (b) Using the abstracts of these papers, we train an embedding model to capture the evolving relationships between these concepts in vector representations over time. In the visualization, gray dots indicate changes in the embedding model’s weights over the years, while the hues of orange, cyan, and red represent the dynamics of word embeddings' parameters as they change with time. (c) The task involves training a machine learning model to predict which currently unconnected concepts (those not yet studied together) are likely to co-occur in the near future, based on the learned embeddings.
• Figure 2: Clustering of Word Embeddings Top panels show clusters generated by the proposed dynamic word embedding method, trained on abstracts from 1994 to 2012 and 2022, respectively. Word embeddings were obtained using a dynamic Word2Vec model trained on the respective set of abstracts. These embeddings were then reduced to two dimensions using UMAP, followed by clustering with a k-means algorithm. The tables below each plot list the key concepts -- by proximity to the clusters center and frequency of occurrence in 2012 (2022) -- identified in each cluster. Clusters generated from independently initialized dimensionality reduction schemes allow for analysis of concept relationships within the same year, however, cluster A0 in 2012 is not directly related to cluster B0 in 2022. Nonetheless, the results demonstrate that the learned word embeddings capture structured information about central topics in the field of quantum physics, illustrating how the landscape of research focus has evolved over the decade.
• Figure 3: Model Confidence: Predictive model trained on the proposed embedding for $\Delta t=[1994,2017]$, tested to predict data in $\Lambda t=[2020,2023]$. (a) Calibration plot showing the probability of the model's predictions being correct, with a comparison to a perfectly calibrated model (orange dashed line). The model is well-calibrated for predictions near probabilities of 0 and 1, confidently classifying these samples. (b) AUC score as a function of the fraction of low-confidence predictions discarded. The plot illustrates that removing uncertain samples enhances the AUC score.
• Figure 4: Validating Past Prediction: Evolution of the prediction probability for three distinct quantum physics concept pairs in a model trained on embeddings up to 2017. Markers indicate the year of the first published abstract containing these concept pairs, as referenced in Refs. Bauman_2019Cincio_2021liu2024simulating2dtopologicalquantum.
• Figure A1: The number of Quantum-Physics Pre-Prints per year is growing: The number of (main) papers publishes on the ArXiv, as well as (inset) the key concepts within them is steadily increasing each year.