Societal citations undermine the function of the science reward system

Abstract

Citations in the scientific literature system do not simply reflect relationships between knowledge but are influenced by non-objective and societal factors. Citation bias, irresponsible citation, and citation manipulation are widespread and have become a serious and growing problem. However, it has been difficult to assess the consequences of mixing societal factors into the literature system because there was no observable literature system unmixed with societal factors for comparison. In this paper, we construct a mathematical theorem network, representing a logic-based and objective knowledge system, to address this problem. By comparing the mathematical theorem network and the scientific citation networks, we find that these two types of networks are significantly different in their structure and function. In particular, the reward function in citation networks is impaired: The scientific citation network fails to provide more recognition for more disruptive results, while the mathematical theorem network can achieve. We develop a network generation model that can create two types of links$\unicode{x2014}$logical and societal$\unicode{x2014}$to account for these differences. The model parameter $q$, which we call the human influence factor, can control the number of societal links and thus regulate the degree of mixing of societal factors in the networks. Under this design, the model successfully reproduces the differences among real networks. These results suggest that the presence of societal factors undermines the function of the scientific reward system. To improve the status quo, we advocate for reforming the reference list format in papers, urging journals to require authors to separately disclose logical references and social references.

Figures (14)

• Figure 1: Scraping citation relationships between mathematical theorems from Metamath web pages. a The Metamath website features a list of mathematical knowledge, including 55 axioms, 760 definitions, 764 syntaxes, and 26,376 theorems. We would only consider theorems and axioms in scraping citation relationships. b Each entry comes with a dedicated page detailing the formal expression of the knowledge and its proof process (the latter only for theorems). This subplot shows part of the webpage for Cauchy's Median Theorem (coded as "cmvth"). c The proof process of Cauchy's Median Theorem is transformed into a network. The ninth step uses the theorem coded as "syl", thus an edge is created between the two nodes "cmvth" and "syl" in the network. The proof is divided into 158 steps and involves 67 theorems. This subplot demonstrates a subgraph induced by these nodes. d The whole data covers a total of ten mathematical fields: logic, set theory, analysis, number theory, algebra, topology, order theory, category theory, geometry, and graph theory. This subplot shows the interdependence for these ten fields, generated using Gephi. The node size is proportional to the number of theorems or axioms in the field, and the edge width is proportional to the number of citations. e Visualization of the large-scale network of all 26,426 nodes, generated with Cosmograph. Five theorems are excluded from the data as they are deprecated in the Metamath project (although their numbers were retained) and constitute isolated nodes in the network.
• Figure 2: Structural differences between the three networks.a--c Cumulative probability distribution of degree of the three networks, by total degree ($k$), out-degree ($k^{out}$), and in-degree ($k^{in}$). These degree distributions have power-law tails, except for the out-degree distribution of the theorem network, which is closer to an exponential distribution. d--f Self-degree correlation of the three networks. The nodes are first binned according to their out-degree, with bin boundaries set at $[3^0, 3^1, 3^2, 3^3, 3^4, 3^5]$ (thus equally spaced in logarithmic coordinates). The average in-degree ($\langle k^{in} \rangle$) of each group is then computed and plotted on a line graph. For instance, the coordinates of the blue dot in the upper leftmost corner of d are $(1, 71.818)$, indicating that the average in-degree for nodes whose out-degree is in the interval $[3^0, 3^1)$ is 71.818. The error bars represent the standard errors. It can be seen that the theorem network and the two citation networks have opposite trends. To further verify this difference, we extracted nodes in the top 20% and last 20% of the out-degree and compared the average in-degree of the two groups across the three networks using the bootstrap method. The insets show the probability distribution of the average in-degree for 1,000 realizations. It can be seen that in the theorem network, nodes in the top 20% by out-degree have a lower average in-degree than those in the last 20%, whereas the opposite is true for the two citation networks. g The average clustering coefficients for the three networks and their corresponding null models. The null model was constructed as outlined in the Methods. The shown values for the null model are the average of the ten null models.
• Figure 3: Functional differences between the three networks.a The average disruption of the three networks and their corresponding null models. They are close to zero in the first two real networks. The distribution of disruption for the three networks can be seen in Extended Data Fig. 2. Since papers have a temporal attribute while theorems do not, we use the concept of topological generations instead of time to calculate disruption, see Methods for details. b Gini coefficient of disruption for the three networks. The disruption was normalised to between 0 and 1, then the Gini coefficient was calculated. The Gini coefficient for the citation network is relatively higher, implying that there is more inequality in the distribution of disruption. c The disruption-citations correlation of the three networks and their corresponding null models. Due to the non-skewed distribution of disruption, we directly use the Pearson correlation coefficient to calculate the correlation between disruption and citations. All three networks weaken the correlation which is under random circumstances, but the reduction in the theorem network is relatively small compared to the citation network. d The disruption-citations correlation of highly cited theorems or papers. In the three networks, the top 10% of cited nodes were selected to calculate the subversion-citation correlation. e--g Using bootstrap methods to observe the difference in average disruption between highly cited and lowly cited nodes. In each of the three networks, the top 20% cited nodes and the last 20% cited nodes were selected, and then the bootstrap method was used to generate 1,000 replications and compute the average disruption to obtain the distribution of average disruption.
• Figure 4: Design of network generation model. The initial network is a subnetwork formed by the 100 nodes that first appeared in the real theorem network. Each time step adds a node to the network, performing the process A shown in the figure. The four parameters included in the model can be explained as follows: (1) $p$: Q.E.D. probability. The repetitive process A stops each time with probability $p$ until it does occur, which can be interpreted as a continuous search for theorems to include in the proof process until the end of the proof. (2) $w$: logical threshold. Determines the range of logistic associations. It affects the number of links $M$ and maximum degree $k_{max}$ of the network. (3) $a$: initial attractiveness of the nodes. This arameter in the preference attachment pattern is used to avoid that a node with $k^{in}=0$ can never be connected. It also affects the $M$ and $k_{max}$ of the network. (4) $q$: human influence factor. Affects the number of societal links. It can measure the degree of influence of societal factors or, in other words, the degree of influence of human behaviour.
• Figure 5: Reproducing the differences using the network generation model. The generative model stops when it reaches the same number of nodes as the real network. The resulting number of links produced succeeds in being similar to the real network. a--c The out-degree distributions of the theorem network generation model and the citation network generation models are significantly different. The former is closer to an exponential distribution, while the latter is closer to a power-law distribution. d The self-degree correlation of the theorem network generation model is negative, while that of the citation network generation models are positive, consistent with reality. The line chart is presented in the same manner as in Fig. \ref{['figure2']}. e--f Gradually increasing clustering coefficients and decreasing disruption-citations correlations from the theorem network generation model, the mathematical citation network generation model, to the cit-HepTh network generation model. The values from the generative model remain close to reality.