Card Sorting with Fewer Cards and the Same Mental Models? A Re-examination of an Established Practice

TL;DR

This paper empirically examines whether presenting a randomized subset of cards (60%) in card sorting preserves the mental models captured by a full card set. Using 160 participants across two domains (e-commerce and banking) and incorporating Big Five personality traits and Cognitive Reflection, the study finds strong alignment in similarity matrices between full and subset sorts, but notable differences in category themes and linguistic labels, especially in more complex content. It documents higher data variability with subsets, necessitating larger sample sizes (about 25–35) and proposes a sample-size model $N=15-90\log\frac{|S\subseteq C|}{|C|}$ to guide planning. The findings offer evidence-based recommendations for employing randomized card subsets while highlighting how study design and individual differences can influence the measurement of mental models, with implications for information architecture research and practice.

Abstract

To keep card sorting with a lot of cards concise, a common strategy for gauging mental models involves presenting participants with fewer randomly selected cards instead of the full set. This is a decades-old practice, but its effects lacked systematic examination. To assess how randomized subsets affect data, we conducted an experiment with 160 participants. We compared results between full and randomized 60\% card sets, then analyzed sample size requirements and the impacts of individual personality and cognitive factors. Our results demonstrate that randomized subsets can yield comparable similarity matrices to standard card sorting, but thematic patterns in categories can differ. Increased data variability also warrants larger sample sizes (25-35 for 60% card subset). Results indicate that personality traits and cognitive reflection interact with card sorting. Our research suggests evidence-based practices for conducting card sorting while exposing the influence of study design and individual differences on measurement of mental models.

Figures (7)

• Figure 1: Card Sorting user interface during the Card Sorting activity. Unsorted cards are on the left while categories containing cards are on the right. Example demonstrates an in-progress sorting of electrical devices.
• Figure 2: Overview of Big Five personality traits (a-e) and Cognitive Reflection (f) score distributions, which are similar between full card set and subset conditions.
• Figure 3: Similarity matrices aggregating card sorting results for each experimental group, demonstrate strong parallels between full set (Full-E and Full-B) and subset (Subset-E and Subset-B) sorting conditions.
• Figure 4: Most frequent standardizations in the e-commerce conditions, based on number of participants who submitted original categories (threshold $\geq$ 4 participants).
• Figure 5: Most frequent standardizations in the banking conditions, based on number of participants who submitted original categories (threshold $\geq$ 4 participants).