Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy

Alberto Baccini, Cristina Re

TL;DR

The paper addresses procedural fairness in research evaluations by arguing that observable panel characteristics alone do not ensure intellectual diversity. It introduces a network-analysis framework using co-authorship, journal-based, and affinity networks to detect hidden connections among panellists. Applying this to Italy's VQR exercises (2004-2010, 2011-2014, 2015-2019), it finds that the ANVUR-appointed panels (2004-2010 and 2011-2014) exhibit markedly higher internal connectivity across all networks than the randomly drawn 2015-2019 control panel, implying unfair intellectual composition. The study highlights the limitations of formal representation criteria and suggests randomization, increased transparency, and network-based diagnostics as practical tools to enhance the fairness and credibility of large-scale research assessments.

Abstract

Research evaluation is usually governed by panels of peers. Procedural fairness refers to the principles that ensures decisions are made through a fair and transparent process. It requires that the composition of panels is fair. A fair panel is usually defined in terms of observable characteristics of scholars such as gender or affiliations. The formal adherence to these criteria is not sufficient to guarantee a fair composition in terms of scholarly thinking, background, or policy orientation. An empirical strategy for exploring the fairness in the intellectual composition of panels is proposed, based on the observation of links between panellists. The case study regards the three panels selected to evaluate research in economics, statistics and business during the Italian research assessment exercises. The first two panels were appointed directly by the governmental agency responsible for the evaluation, while the third was randomly selected. Hence the third panel can be considered as a control for evaluating about the fairness of the others. The fair representation is explored by comparing the networks of panellists based on their co-authorship relations, the networks based on journals in which they published and the networks based on their affiliated institutions (universities, research centres and newspapers). The results show that the members of the first two panels had connections much higher than the members of the control group. Hence the composition of the first two panels should be considered as unfair, as the results of the research assessments.

Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy

TL;DR

The paper addresses procedural fairness in research evaluations by arguing that observable panel characteristics alone do not ensure intellectual diversity. It introduces a network-analysis framework using co-authorship, journal-based, and affinity networks to detect hidden connections among panellists. Applying this to Italy's VQR exercises (2004-2010, 2011-2014, 2015-2019), it finds that the ANVUR-appointed panels (2004-2010 and 2011-2014) exhibit markedly higher internal connectivity across all networks than the randomly drawn 2015-2019 control panel, implying unfair intellectual composition. The study highlights the limitations of formal representation criteria and suggests randomization, increased transparency, and network-based diagnostics as practical tools to enhance the fairness and credibility of large-scale research assessments.

Abstract

Research evaluation is usually governed by panels of peers. Procedural fairness refers to the principles that ensures decisions are made through a fair and transparent process. It requires that the composition of panels is fair. A fair panel is usually defined in terms of observable characteristics of scholars such as gender or affiliations. The formal adherence to these criteria is not sufficient to guarantee a fair composition in terms of scholarly thinking, background, or policy orientation. An empirical strategy for exploring the fairness in the intellectual composition of panels is proposed, based on the observation of links between panellists. The case study regards the three panels selected to evaluate research in economics, statistics and business during the Italian research assessment exercises. The first two panels were appointed directly by the governmental agency responsible for the evaluation, while the third was randomly selected. Hence the third panel can be considered as a control for evaluating about the fairness of the others. The fair representation is explored by comparing the networks of panellists based on their co-authorship relations, the networks based on journals in which they published and the networks based on their affiliated institutions (universities, research centres and newspapers). The results show that the members of the first two panels had connections much higher than the members of the control group. Hence the composition of the first two panels should be considered as unfair, as the results of the research assessments.
Paper Structure (9 sections, 2 figures, 22 tables)

This paper contains 9 sections, 2 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Research evaluation, procedural fairness and panel composition
  3. The Italian research assessment exercises and the selection of panellists.
  4. Research design
  5. The co-authorship networks
  6. The journal based networks
  7. The affinity networks
  8. Discussion and conclusions
  9. The panellists selection process

Figures (2)

  • Figure A1: The largest component in the co-authorship network for the panel 2004-2010. Size of nodes is proportional to betweenness centrality. Red nodes are panel members.
  • Figure A2: The largest component of the co-authorship network for the panel 2011-2014. Size of vertices is proportional to betweenness centrality. Red nodes are panel members.