How to predict creativity ratings from written narratives: A comparison of co-occurrence and textual forma mentis networks

TL;DR

The paper addresses predicting creativity ratings from short narratives by comparing two semantic-network representations: surface-based co-occurrence networks and dependency-based textual forma mentis networks (TFMNs). It provides a step-by-step workflow to preprocess text, construct networks, extract structural, spreading-activation, and emotion features, and train regression models with cross-validation. Empirical results on 1029 stories show that TF MNs consistently outperform co-occurrence models, with network structural features driving predictive power and emotions offering modest improvements; spreading-activation adds little beyond topology. The study delivers practical, open, and interpretable methods for applying network-based text analysis in creativity research and related cognitive domains, with broad implications for reproducible workflow design and AI-assisted narrative analysis.

Abstract

This tutorial paper provides a step-by-step workflow for building and analysing semantic networks from short creative texts. We introduce and compare two widely used text-to-network approaches: word co-occurrence networks and textual forma mentis networks (TFMNs). We also demonstrate how they can be used in machine learning to predict human creativity ratings. Using a corpus of 1029 short stories, we guide readers through text preprocessing, network construction, feature extraction (structural measures, spreading-activation indices, and emotion scores), and application of regression models. We evaluate how network-construction choices influence both network topology and predictive performance. Across all modelling settings, TFMNs consistently outperformed co-occurrence networks through lower prediction errors (best MAE = 0.581 for TFMN, vs 0.592 for co-occurrence with window size 3). Network-structural features dominated predictive performance (MAE = 0.591 for TFMN), whereas emotion features performed worse (MAE = 0.711 for TFMN) and spreading-activation measures contributed little (MAE = 0.788 for TFMN). This paper offers practical guidance for researchers interested in applying network-based methods for cognitive fields like creativity research. we show when syntactic networks are preferable to surface co-occurrence models, and provide an open, reproducible workflow accessible to newcomers in the field, while also offering deeper methodological insight for experienced researchers.

Figures (9)

• Figure 1: Overview of the analysis pipeline. Raw stories are lemmatised and tokenised (Steps 1–2), then converted into three network types (Step 3). TFMNs also capture the valence of words, which is encoded as red for negative, cyan for positive, and grey for neutral. From each network, we extract structural measures, stationary spreading-activation values seeded by the prompt words, and emotion scores from the raw text (Step 4). Regression models are then trained on four feature sets to predict human creativity ratings (Step 5).
• Figure 2: Story: "There is a heavy gloom hanging over me today. The loan sharks are after me because I have a payment that is late. They exist to punish people like me, down own there luck. I might as well face the music and ask the boss for an extension on paying him back even though it will cost me more. I have learned one thing though, never bet on a rubber ducky race and use a bookie." The panels show co-occurrence networks without pronouns (top), with pronouns (middle), and the textual forma mentis network (bottom). The final panels show ASPL and clustering coefficient distributions for WS4, including TFMN for comparison.
• Figure 3: Predictive performance across models and network builders. Note. Panel (a) shows mean absolute error (MAE), where lower values indicate better predictive accuracy. Panel (b) shows Spearman rank correlations, where higher values indicate stronger association with human creativity ratings. Models (rows) and builders (columns) are ordered according to the MAE rank (lowest to highest).
• Figure 4: SHAP beeswarm plots for the best-performing models. Note. The left panel corresponds to the Gradient Boosting model trained on the TFMN representation. The right panel corresponds to the XGBoost model trained on the pronoun-inclusive co-occurrence network with window size 4. Colours indicate feature values, and horizontal position reflects SHAP impact on predicted creativity scores.
• Figure 5: Rater-specific SHAP beeswarm plots for the best-performing model--builder pair per rater. Panels show H (top-left; coocc_p_WS4 with SGD regression), J (top-right; coocc_p_WS4 with linear regression), K (bottom-left; TFMN with Gradient Boosting), and N (bottom-right; TFMN with Random Forest). Colours indicate feature values, and horizontal position reflects SHAP impact on predicted creativity scores.