Modeling within-department homogeneity in research quality rankings: an application to the Italian ISPD

Abstract

In this paper, we consider the academic department ranking system of Italy, which is based on a performance index named Indice Standardizzato di Performance Dipartimentale (ISPD). While critiques to the ISPD have been moved for its marked tendency to polarization, we here formalize a yet unexplored determinant of this phenomenon, that is, the presence of within-department homogeneity among the standardized scores used to build the index. We account for this intra-departmental correlation by modeling it as a function of departments' size. The proposed model, estimated via Maximum Likelihood, allows to build a fairer ranking procedure via the definition of a properly adjusted version of the ISPD. The estimation framework is also adapted to fit publicly available data, which are coarsened by rounding and/or left-truncated. To this end, a novel probability distribution termed Betoidal is introduced. Empirical evidence in favor of the proposed model is found in the 2017 and 2022 data. Moreover, a simulation study shows that the adjusted index significantly overcomes not only the original ISPD, but also other more data-demanding competing proposals.

Figures (6)

• Figure 1: Distribution of ISPDs for ANVUR's (a) 2017 ranking exercise (766 departments) and (b) 2022 exercise (top 350 departments only).
• Figure 2: Two settings with $\rho_{\textup{A}}=0.05=\rho_{\textup{B}}$, $N_{\textup{A}}=75$, $N_{\textup{B}}=150$, and (a) $z_{\textup{A}}=2=z_{\textup{B}}$, and (b) $z_{\textup{A}}=-2=z_{\textup{B}}$. In (a), $\textup{ISPD}_{\textup{A}}=97.72\%=\textup{ISPD}_{\textup{B}}$, with the correct performance measures being $P(Z_{\textup{A}} \leq 2)=82.19\%$ and $P(Z_{\textup{B}} \leq 2)=75.43\%$. In (b), $\textup{ISPD}_{\textup{A}}=2.28\%=\textup{ISPD}_{\textup{B}}$, with $P(Z_{\textup{A}} \leq -2)=17.81\%$ and $P(Z_{\textup{B}} \leq -2)=24.57\%$.
• Figure 3: (a) Betoidal density for various levels of $\sigma$, and (b) relation between $\sigma$ and $a$ for two random variables $X\sim \textup{Betoidal}(\sigma)$ and $Y\sim\textup{Beta}(a,a)$ with $V(X)=V(Y)$.
• Figure 4: Density of $X\sim \textup{Betoidal}(\sigma)$ and $Y\sim\textup{Beta}(a,a)$ with $V(X)=V(Y)$ and (a) $\sigma=0.5$ ($a=3.4005$), and (b) $\sigma=2.5$ ($a=0.2568$).
• Figure 5: Empirical histograms of (a) $\text{ISPD}$ and (b) $\text{ISPD}^{\textsc{fcm}}$ for the medium perturbation scenario (all departments and datasets combined).