Analyzing-Evaluating-Creating: Assessing Computational Thinking and Problem Solving in Visual Programming Domains

TL;DR

A novel test focusing on the three higher cognitive levels in Bloom's Taxonomy, i.e., Analyze, Evaluate, and Create, based on multi-choice items enabling psychometric validation and usage in large-scale studies is developed.

Abstract

Computational thinking (CT) and problem-solving skills are increasingly integrated into K-8 school curricula worldwide. Consequently, there is a growing need to develop reliable assessments for measuring students' proficiency in these skills. Recent works have proposed tests for assessing these skills across various CT concepts and practices, in particular, based on multi-choice items enabling psychometric validation and usage in large-scale studies. Despite their practical relevance, these tests are limited in how they measure students' computational creativity, a crucial ability when applying CT and problem solving in real-world settings. In our work, we have developed ACE, a novel test focusing on the three higher cognitive levels in Bloom's Taxonomy, i.e., Analyze, Evaluate, and Create. ACE comprises a diverse set of 7x3 multi-choice items spanning these three levels, grounded in elementary block-based visual programming. We evaluate the psychometric properties of ACE through a study conducted with 371 students in grades 3-7 from 10 schools. Based on several psychometric analysis frameworks, our results confirm the reliability and validity of ACE. Our study also shows a positive correlation between students' performance on ACE and performance on Hour of Code: Maze Challenge by Code.org.

Figures (26)

• Figure 1: (a) shows the distribution of test items w.r.t to CT and problem-solving concepts and Bloom's cognitive levels. (b)--(f) are examples of five items from ACE. These items are grounded in the domain of Hour of Code: Maze Challenge (HoCMaze) hourofcode_maze, which can be found at studio.code.org/s/hourofcode. HoCMaze domain comprises elementary block-based visual programming tasks where one has to write a solution code that would navigate the Avatar (blue dart) to the Goal (red star) without crashing into Walls (gray grid cells). We encourage the reader to attempt these items; all $21$ test items from ACE along with their answers are provided in the appendix.
• Figure 2: An overview of the performance of students on ACE. (a) overall distribution of ACE scores across all $371$ students; (b) distribution of ACE score per grade; (c) success rate of students for each item in ACE. Details are in Section \ref{['sec.setup']}.
• Figure 3: Results from a 1-parameter Rasch model rasch1993probabilistic on the ACE items and student scores. (a) Item characteristic curve for each item in ACE and (b) Wright map corresponding to our student population.
• Figure 4: Q17. Avatar design
• Figure 5: Pearson's correlation coefficient, $r$, between ACE and HoCMaze, between ACE and its categories, and between each category. All values are significant with $p<0.001$.