How to quantify an examination? Evidence from physics examinations via complex networks

TL;DR

This study introduces knowledge point networks (KPNs) as a quantitative framework to analyze examination structure, applying it to 35 physics NCEE exams from 2006–2020. The authors show that KPNs are predominantly scale-free, assortative, and small-world, with clear communities centered on mechanics and electromagnetism, and they identify core knowledge points via eigenvector centrality. They propose a comprehensive difficulty coefficient $F_d = \langle k \rangle \cdot \rho \cdot T \cdot \langle c \rangle$ that correlates with actual exam difficulty and demonstrate how KPN topology evolves with curriculum reform and across volumes and question types. The findings offer objective tools for exam design and teaching adjustments, and they point to future work extending the framework to directed and weighted networks, other subjects, and broader educational contexts.

Abstract

Given the untapped potential for continuous improvement of examinations, quantitative investigations of examinations could guide efforts to considerably improve learning efficiency and evaluation and thus greatly help both learners and educators. However, there is a general lack of quantitative methods for investigating examinations. To address this gap, we propose a new metric via complex networks; i.e., the knowledge point network (KPN) of an examination is constructed by representing the knowledge points (concepts, laws, etc.) as nodes and adding links when these points appear in the same question. Then, the topological quantities of KPNs, such as degree, centrality, and community, can be employed to systematically explore the structural properties and evolution of examinations. In this work, 35 physics examinations from the NCEE examination spanning from 2006 to 2020 were investigated as an evidence. We found that the constructed KPNs are scale-free networks that show strong assortativity and small-world effects in most cases. The communities within the KPNs are obvious, and the key nodes are mainly related to mechanics and electromagnetism. Different question types are related to specific knowledge points, leading to noticeable structural variations in KPNs. Moreover, changes in the KPN topology between examinations administered in different years may offer insights guiding college entrance examination reforms. Based on topological quantities such as the average degree, network density, average clustering coefficient, and network transitivity, the Fd is proposed to evaluate examination difficulty. All the above results show that our approach can comprehensively quantify the knowledge structures and examination characteristics. These networks may elucidate comprehensive examination knowledge graphs for educators and guide improvements in teaching.

Figures (7)

• Figure 1: Several important network results of KPNs.a, The r-statistics of 35 KPNs. Only 11V2 has a negative R value. b, Degree distribution of IKPN nodes in the integrated network. The degree distribution of the IKPN exhibits a pronounced fat-tail distribution. c, The small-coefficient $\sigma$ is defined as $\sigma = \frac{\left( \frac{C_c}{C_r} \right)}{\left( \frac{L_c}{L_r} \right)}$, where $C_c$ represents the average clustering coefficient of the largest connected subgraph, $C_r$ represents the average clustering coefficient of the corresponding ER network, and $L_r$ is the average shortest path length for the random network. As per the theoretical framework of the small-world effect, if a network's average path length closely resembles that of a random network of equivalent size, and the network's clustering coefficient significantly surpasses that of the corresponding random graph, i.e., $\sigma$ > 1 ($C_c/C_r \gg 1$ ,and $L_c/L_r \geq 1$) , then the network can be considered a small-world network. d, 20V1 and its corresponding random network graph. It is evident that the network of 20V1 exhibits stronger clustering than its random network. e, Number of communities and modularity values(Q) of the network. Number of blocks represents the number of communities excluding isolated nodes, pairs of nodes and non-closed triangular communities (The three types of communities encompass fewer knowledge points and can roughly represent the number of simple questions in each exam) from the network, reflecting more complex communities. f, Monotonicity values of 4 centrality indexes in 35 Networks. The red curve representing the Eigenvector Centrality $E$ exhibits the highest monotonicity.
• Figure 1: The KPN of 12V2 is divided into different colors according to the community. The KPN primarily consists of three dual-node communities, four triangular communities, three complex communities, and one isolated node community.
• Figure 2: The largest community graph representing the IKPN. The node size is determined according to the degree value. The red nodes represent knowledge related to mechanics, the blue nodes represent knowledge related to electromagnetism, and the purple nodes represent knowledge related to physical optics. The color distinctions in the figure underscore the pronounced clustering features evident within the network.
• Figure 2: The distribution diagram of the top 15 key knowledge points among the 35 KPNs. In this diagram, orange represents knowledge points related to mechanics, red represents knowledge points related to electromagnetics, blue represents knowledge points related to thermodynamics, green represents knowledge points related to optics, and yellow represents knowledge points related to atomic physics. The ranking of key nodes is based on eigenvector centrality ($E$), with the diagram being sorted in descending order. Thus, the 15th position on the vertical axis corresponds to the node with the highest eigenvector centrality, ranked first.
• Figure 3: Several results of IKPN across different periods, volumes, and question types.a, Network diagrams of the IKPNs for the four stages. Node sizes are determined according to $E$. As the E value increases, the nodes become larger, and their color approaches red. The second stage exhibits strong local clustering effects. The overall network density of the first, third, and fourth stages continues to increase. b, Network diagrams with different volumes. Node sizes are determined according to $E$. V1 is significantly denser than V2, with fewer simple communities and stronger clustering effects. c, The degree distribution graphs for both volumes, V1 KPN and V2 KPN, exhibit scale-free characteristics with a clear fat-tailed distribution. d, Comparative diagrams of the difficulty in two volumes. The difficulty of V1 is consistently higher than V2 for most years, with the exception of the period between 2009 and 2012, during which V1 was less difficult than V2. In terms of quantity, V1 has consistently been proven to be more challenging than V2. Moreover, this characteristic became more pronounced and stable after 2013. e, The network diagrams of the four versions of KPN based on question types. Node sizes are sorted according to the Eigenvector Centrality index $E$. The preferred knowledge points of the four KPNs differ, leading to distinct differences in their network diagrams. Choice question contains the most nodes, while calculation question exhibits the most frequent connections between nodes.