Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures

Shan Zhang, Ruiwei Xiao, Anthony F. Botelho, Guanze Liao, Thomas K. F. Chiu, John Stamper, Kenneth R. Koedinger

TL;DR

This study addresses the need for psychometrically validated measures of K-12 teachers' AI literacy by developing and validating both self-reported and objective-based instruments within the NG 2021 framework, and by examining their alignment through latent-profile analyses. Using Rasch analysis, CFA, and LPA on 288 Taiwanese teachers (with and without prior AI literacy experience), the authors establish a robust OB three-factor structure and a four-factor SR structure, while revealing consistently weak correlations between SR and OB. Six comprehensive SR–OB profiles emerge for the full sample, with subgroup-specific patterns showing that prior AI literacy experience mitigates extreme misalignment and that a unique low–low pattern occurs among those without prior exposure. The work demonstrates how SR and OB measures can be integrated to form calibration-aware learner models, informing diagnostic PD and scalable learning analytics interventions, and it emphasizes Ethics as a core dimension in both measurement streams. Overall, the study contributes theoretically by extending AI-literacy frameworks to include validated SR–OB constructs on shared dimensions and practically by enabling diagnostic, adaptive PD tools that support targeted teacher growth and responsible AI teaching practices.

Abstract

The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.

How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures

TL;DR

This study addresses the need for psychometrically validated measures of K-12 teachers' AI literacy by developing and validating both self-reported and objective-based instruments within the NG 2021 framework, and by examining their alignment through latent-profile analyses. Using Rasch analysis, CFA, and LPA on 288 Taiwanese teachers (with and without prior AI literacy experience), the authors establish a robust OB three-factor structure and a four-factor SR structure, while revealing consistently weak correlations between SR and OB. Six comprehensive SR–OB profiles emerge for the full sample, with subgroup-specific patterns showing that prior AI literacy experience mitigates extreme misalignment and that a unique low–low pattern occurs among those without prior exposure. The work demonstrates how SR and OB measures can be integrated to form calibration-aware learner models, informing diagnostic PD and scalable learning analytics interventions, and it emphasizes Ethics as a core dimension in both measurement streams. Overall, the study contributes theoretically by extending AI-literacy frameworks to include validated SR–OB constructs on shared dimensions and practically by enabling diagnostic, adaptive PD tools that support targeted teacher growth and responsible AI teaching practices.

Abstract

The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.
Paper Structure (29 sections, 5 figures)

This paper contains 29 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Literacy For K-12 Educators
  4. Professional Development for AI Literacy in K–12
  5. Measuring Teachers' AI Literacy using Both Self-Report- and Objective Measure-Based Assessment
  6. Methods
  7. Survey Design
  8. Objective-Based Items
  9. Self-Report Items
  10. Demographics Items
  11. Data Collection and Participants
  12. Data Processing and Analysis
  13. Examining Psychometric Stability
  14. Factor Analyses
  15. Latent Profile Analysis
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Wright Map of the AI Literacy Objective Measure (1PL Rasch Model).
  • Figure 2: Confirmatory Factor Analysis of the AI Literacy Objective Measure with Three Factors
  • Figure 3: Confirmatory Factor Analysis of the AI Literacy Self-Report Measure with Four Factors
  • Figure 4: Latent Profile Analysis of Combined Self-Reported and Objective-Based AI Literacy (N=288)
  • Figure 5: Latent Profile Analyses of combined self-reported and objective-based AI literacy by prior AI-literacy experience.