What is missing from this picture? Persistent homology and mixup barcodes as a means of investigating negative embedding space

TL;DR

This paper addresses how negative embedding space in topic model embeddings can reflect missing context and potential interdisciplinarity. It combines persistent homology to identify holes in the embedding landscape with mixup barcodes to quantify how unobserved publications fill those holes, using top2vec embeddings of dimensions.ai UF data. The study finds $252$ persistent holes, and shows that older pretraining documents more often fill these holes than newer posttraining documents, with hole specific patterns revealing interdisciplinary fillings. The results suggest negative space encodes historical context and selective recombination, offering a framework for understanding conceptual landscapes, guiding pretraining strategies, and informing science policy and funding decisions about interdisciplinarity and missing context.

Abstract

Recent work in the information sciences, especially informetrics and scientometrics, has made substantial contributions to the development of new metrics that eschew the intrinsic biases of citation metrics. This work has tended to employ either network scientific (topological) approaches to quantifying the disruptiveness of peer-reviewed research, or topic modeling approaches to quantifying conceptual novelty. We propose a combination of these approaches, investigating the prospect of topological data analysis (TDA), specifically persistent homology and mixup barcodes, as a means of understanding the negative space among document embeddings generated by topic models. Using top2vec, we embed documents and topics in n-dimensional space, we use persistent homology to identify holes in the embedding distribution, and then use mixup barcodes to determine which holes are being filled by a set of unobserved publications. In this case, the unobserved publications represent research that was published before or after the data used to train top2vec. We investigate the extent that negative embedding space represents missing context (older research) versus innovation space (newer research), and the extend that the documents that occupy this space represents integrations of the research topics on the periphery. Potential applications for this metric are discussed.

Figures (10)

• Figure 1: Left: the collection of black dots is called a point cloud. Middle: consider balls of a fixed radius centered at each point. If two balls intersect, join them by an edge. If three balls have a common intersection, fill the triangle between the three edges arising from the pairwise intersections. Similarly, add higher-dimensional simplices for higher-order intersections. Right: the simplicial complex (consisting of vertices, edges, triangles, and higher-order simplices) obtained from this construction.
• Figure 2: In (a), we recovered the homological information of a point cloud data which resembles an $8$-figure with persistent homology. The two colored bars correspond to two holes surrounded by a $1$-dimensional boundary. In (b), the persistence barcode is represented as a multi-set of points called a persistence diagram. For the green point in the persistence barcode, a representative cycle is visualized as the green cycle in (c).
• Figure 3: Point cloud $P$ contains the black dots, and point cloud $Q$ contains the red crosses. (a) Point cloud $P$. (b) Point clouds $P$ and $Q$. Using standard persistent homology we can find holes in dimension $1$ for point cloud $P$. In (c), persistence barcode with the two longest bars for the $1$st homology group of point cloud $P$ is visualized. In (d), representative cycle of the longest bar (green color) is visualized in point cloud $P$ and $Q$. Now, we consider how the inclusion of point cloud $Q$ effect the hole visualized by the representative cycle in green color. (e) Growing balls around point cloud $P$ and $Q$ fills the hole visualized by the green representative cycle earlier compared to when we just have point cloud $P$. (f) The yellow bar represents how the inclusion of point cloud $Q$ into point cloud $P$ has reduced the length of the original green bar.
• Figure 4: (a) Persistence diagram in degree $1$ with a feature highlighted in red for topics embeddings obtained after projection into 2d using UMAP. (b) Extracted representative cycles plotted on the topics embedding for the highlighted feature in red. (c) Representative cycle with topic index. (d) Subject classification for different topics. (e) Topic information for topic indices which are part of the representative cycle.
• Figure 5: In (a), training data are the articles which were used to generate topics. Classes A and B are the articles, some of which were added later for analysis. Total mixup is calculated in 100 iterations for each class. In one iteration, 10 percent of randomly sampled points from a class are added to topics embedding, and total mixup was recorded. Histogram (b) of total mixups. Box plot (c) represents the distribution of total mixup for each class. P-value is calculated using permutation test, where the observed test statistics (absolute difference between means) was compared with 10,000 random permutations of the dataset.