AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupations

TL;DR

AI literacy definitions often emphasize technical knowledge at the expense of workplace use. The authors develop a work-task oriented AI literacy assessment and test it in a US Navy robotics training program, combining two training iterations with a scenario-based competition. They adapt Hornberger's AI-LIT-H, revise AI-LIT-MH, and introduce COMP-MCQ, but find that traditional literacy scores do not reliably track gains or predict competition performance, though the COMP-MCQ shows meaningful correlations with applied tasks. The study argues for domain-specific, scenario-based assessments that reflect authentic job decisions, highlighting the value of integrated training and evaluation approaches for AI-enabled work.

Abstract

As artificial intelligence (AI) systems become ubiquitous in professional contexts, there is an urgent need to equip workers, often with backgrounds outside of STEM, with the skills to use these tools effectively as well as responsibly, that is, to be AI literate. However, prevailing definitions and therefore assessments of AI literacy often emphasize foundational technical knowledge, such as programming, mathematics, and statistics, over practical knowledge such as interpreting model outputs, selecting tools, or identifying ethical concerns. This leaves a noticeable gap in assessing someone's AI literacy for real-world job use. We propose a work-task-oriented assessment model for AI literacy which is grounded in the competencies required for effective use of AI tools in professional settings. We describe the development of a novel AI literacy assessment instrument, and accompanying formative assessments, in the context of a US Navy robotics training program. The program included training in robotics and AI literacy, as well as a competition with practical tasks and a multiple choice scenario task meant to simulate use of AI in a job setting. We found that, as a measure of applied AI literacy, the competition's scenario task outperformed the tests we adopted from past research or developed ourselves. We argue that when training people for AI-related work, educators should consider evaluating them with instruments that emphasize highly contextualized practical skills rather than abstract technical knowledge, especially when preparing workers without technical backgrounds for AI-integrated roles.

Figures (2)

• Figure 1: Neither AI-LIT-H nor AI-LIT-MH increased after AI User training.
• Figure 2: Both AI-LIT-MH and COMP-MCQ instruments had a significant relationship with performance on the AI User training.