AI-enhanced semantic feature norms for 786 concepts

TL;DR

Semantic feature norms are foundational for modeling human conceptual knowledge but are costly to generate with limited coverage. The authors introduce NOVA, an AI-enhanced norm dataset that combines human elicitation with AI-assisted verification, built for 786 concepts and enabling a final set with approximately 750 concepts and over 8,000 features. NOVA exhibits far higher feature density per concept (approximately 700) and greater feature overlap than human-only norms, while preserving meaningful category structure, and it better predicts human semantic similarity than both human norms and word embeddings, as shown in a triadic similarity task where NOVA achieved 86.20% agreement versus 60.40% for FastText (p<0.001). This work demonstrates that carefully validated LLM-based augmentation can enrich cognitive science data and offers a replicable workflow for scalable, human-aligned norm generation that can inform neural and computational models of semantic memory.

Abstract

Semantic feature norms have been foundational in the study of human conceptual knowledge, yet traditional methods face trade-offs between concept/feature coverage and verifiability of quality due to the labor-intensive nature of norming studies. Here, we introduce a novel approach that augments a dataset of human-generated feature norms with responses from large language models (LLMs) while verifying the quality of norms against reliable human judgments. We find that our AI-enhanced feature norm dataset, NOVA: Norms Optimized Via AI, shows much higher feature density and overlap among concepts while outperforming a comparable human-only norm dataset and word-embedding models in predicting people's semantic similarity judgments. Taken together, we demonstrate that human conceptual knowledge is richer than captured in previous norm datasets and show that, with proper validation, LLMs can serve as powerful tools for cognitive science research.

Figures (5)

• Figure 1: A schematic representation of our workflow. Features were initially crowd-sourced for 786 concepts, forming a human-generated matrix. A subset of 10,000 concept-feature pairs underwent validation via human judgments. LLM responses were compared to these human judgments to determine the best-performing strategy. Using this method, LLMs completed the matrix for all 8,200 selected features, forming the AI-augmented matrix.
• Figure 2: Models' ability to reliably predict human feature-concept ratings measured as $d'$ using raw responses (orange) and responses re-verified using GPT-4o. Bar heights show mean $d'$ across the 0-shot and 2-shot experiments. Gray and black dashed lines correspond to GPT-4o's performance in the 0-shot and 2-shot settings respectively. Errorbars correspond to bootstrapped 95% confidence intervals.
• Figure 3: $t$-stochastic neighbor embeddings of the semantic vectors for each of 786 concepts derived from the final verified matrix. Category labels were generated by combining higher order labels from existing norm datasets and LLM-suggested categories from GPT-4o.
• Figure 4: (A) Counts of valid features per concept and number of concepts that share common features for the reduced human-generated matrix (top row) and AI-enhanced norm matrix (bottom row). (B) Pairwise cosine dissimilarity matrices based on the reduced human-generated norm matrix (left) and AI-enhanced norm matrix (right).
• Figure 5: (A) Procedure for generating trials for the triadic judgment experiment and an example trial. (B) Proportion of human responses that aligned with the human matrix (yellow bar) vs. the AI-enhanced matrix (purple bar) and with FastText word embeddings (orange) vs. AI-enhanced semantic vectors (purple) in Experiment 2. Error bars represent standard errors of the means.