Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GLAT: The Generative AI Literacy Assessment Test

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević, Lixiang Yan

TL;DR

The results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future.

Abstract

The rapid integration of generative artificial intelligence (GenAI) technology into education necessitates precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased. In this study, we present the GenAI Literacy Assessment Test (GLAT), a 20-item multiple-choice instrument developed following established procedures in psychological and educational measurement. Structural validity and reliability were confirmed with responses from 355 higher education students using classical test theory and item response theory, resulting in a reliable 2-parameter logistic (2PL) model (Cronbach's alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97). Critically, GLAT scores were found to be significant predictors of learners' performance in GenAI-supported tasks, outperforming self-reported measures such as perceived ChatGPT proficiency and demonstrating external validity. These results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future.

GLAT: The Generative AI Literacy Assessment Test

TL;DR

The results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future.

Abstract

The rapid integration of generative artificial intelligence (GenAI) technology into education necessitates precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased. In this study, we present the GenAI Literacy Assessment Test (GLAT), a 20-item multiple-choice instrument developed following established procedures in psychological and educational measurement. Structural validity and reliability were confirmed with responses from 355 higher education students using classical test theory and item response theory, resulting in a reliable 2-parameter logistic (2PL) model (Cronbach's alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97). Critically, GLAT scores were found to be significant predictors of learners' performance in GenAI-supported tasks, outperforming self-reported measures such as perceived ChatGPT proficiency and demonstrating external validity. These results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future.

Paper Structure

This paper contains 33 sections, 1 equation, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. AI and GenAI Literacy
  4. AI Literacy Instruments
  5. Classical Test Theory and Item Response Theory
  6. External Validity and Domain Knowledge
  7. Generative AI Literacy Assessment Test (GLAT)
  8. Methods
  9. Participants
  10. Item Generation
  11. Blueprint construction
  12. Item generation
  13. Content validity.
  14. RQ1: Structural Validity and Reliability
  15. Item selection
  16. ...and 18 more sections

Figures (6)

  • Figure 1: The participant sample size and focus of each validation study.
  • Figure 3: Visual analytics on teamwork in healthcare simulations, including: a) a bar chart of four prioritisation strategies, b) a social network diagram of communication behaviours among the actors, and c) a ward map showing individuals' physical positions (hexagon), verbal communication duration (colour saturation), and peak heart rate locations.
  • Figure 4: System design of the generative AI (GenAI) chatbots.
  • Figure 5: Study design: three literacy measurements, a baseline task, and an AI-assisted task.
  • Figure 6: Item characteristic curves for the Rasch, 2PL, and 3PL models. $\theta$ presents the latent trait, GenAI literacy.
  • ...and 1 more figures