Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach

Francesca Mangili, Giorgia Adorni, Alberto Piatti, Claudio Bonesana, Alessandro Antonucci

TL;DR

The paper addresses automatic learner-competence assessment by deriving a Bayesian-network-based model directly from an assessment rubric, enabling probabilistic inference from observed behaviours. It introduces a Noisy-OR BN framework to maintain a compact parameter set and to reflect partial rubric ordering. The authors translate a rubric into a BN with latent cells $X_{rc}$ and task observations $Y^t_{rc}$, validating the approach on a K-12 Cross Array Task dataset and reporting a high correlation of $r=0.94$ between BN-based scores and expert rubric scores, along with interpretable posterior profiles. The approach supports rapid, adaptive assessment in ITS and offers a practical path for rubric-driven automatic assessment with real-time inference.

Abstract

Automatic assessment of learner competencies is a fundamental task in intelligent tutoring systems. An assessment rubric typically and effectively describes relevant competencies and competence levels. This paper presents an approach to deriving a learner model directly from an assessment rubric defining some (partial) ordering of competence levels. The model is based on Bayesian networks and exploits logical gates with uncertainty (often referred to as noisy gates) to reduce the number of parameters of the model, so to simplify their elicitation by experts and allow real-time inference in intelligent tutoring systems. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills. The simple elicitation of the model starting from the assessment rubric opens up the possibility of quickly automating the assessment of several tasks, making them more easily exploitable in the context of adaptive assessment tools and intelligent tutoring systems.

Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach

TL;DR

The paper addresses automatic learner-competence assessment by deriving a Bayesian-network-based model directly from an assessment rubric, enabling probabilistic inference from observed behaviours. It introduces a Noisy-OR BN framework to maintain a compact parameter set and to reflect partial rubric ordering. The authors translate a rubric into a BN with latent cells and task observations , validating the approach on a K-12 Cross Array Task dataset and reporting a high correlation of between BN-based scores and expert rubric scores, along with interpretable posterior profiles. The approach supports rapid, adaptive assessment in ITS and offers a practical path for rubric-driven automatic assessment with real-time inference.

Abstract

Automatic assessment of learner competencies is a fundamental task in intelligent tutoring systems. An assessment rubric typically and effectively describes relevant competencies and competence levels. This paper presents an approach to deriving a learner model directly from an assessment rubric defining some (partial) ordering of competence levels. The model is based on Bayesian networks and exploits logical gates with uncertainty (often referred to as noisy gates) to reduce the number of parameters of the model, so to simplify their elicitation by experts and allow real-time inference in intelligent tutoring systems. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills. The simple elicitation of the model starting from the assessment rubric opens up the possibility of quickly automating the assessment of several tasks, making them more easily exploitable in the context of adaptive assessment tools and intelligent tutoring systems.
Paper Structure (8 sections, 5 equations, 3 figures, 2 tables)

This paper contains 8 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Exploiting Noisy Gates in BN-based Learner Models
  3. BN-based learner model
  4. Noisy-OR
  5. Translating Assessment Rubric into BNs
  6. A Case Study on K-12 Algorithmic Skills
  7. Results
  8. Conclusions

Figures (3)

  • Figure 1: Example of BN-based learner model. Adapted from Anonymous2022.
  • Figure 2: A noisy gate (explicit formulation).
  • Figure 3: Cross array schemes (top) and corresponding parameters (bottom). The rows represent answers, columns the skills, and darker shades of grey lower skill-answer inhibition probabilities (white for non-relevant skills).