S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment

TL;DR

The paper tackles the challenge of measuring divergent thinking (DT) across languages and cultures by introducing S-DAT, a multilingual, GenAI-driven framework that computes semantic distance using transformer-based embeddings. It selects Granite-embedding-278m-multilingual for robust cross-lingual scoring, calibrates scores to the original DAT distribution, and validates the approach against human creativity measures including the AUT and Bridge-the-Associative-Gap Task. Key contributions include achieving cross-linguistic stability without language-specific calibrations, providing percentile-based interpretation, and demonstrating convergent but discriminant validity relative to convergent thinking tasks. The work enables scalable, fair, online DT assessment suitable for global-scale creativity research and applications, while acknowledging limitations in surface language scripts and the need for potential hybrid scoring approaches to capture broader creativity facets.

Abstract

This paper introduces S-DAT (Synthetic-Divergent Association Task), a scalable, multilingual framework for automated assessment of divergent thinking (DT) -a core component of human creativity. Traditional creativity assessments are often labor-intensive, language-specific, and reliant on subjective human ratings, limiting their scalability and cross-cultural applicability. In contrast, S-DAT leverages large language models and advanced multilingual embeddings to compute semantic distance -- a language-agnostic proxy for DT. We evaluate S-DAT across eleven diverse languages, including English, Spanish, German, Russian, Hindi, and Japanese (Kanji, Hiragana, Katakana), demonstrating robust and consistent scoring across linguistic contexts. Unlike prior DAT approaches, the S-DAT shows convergent validity with other DT measures and correct discriminant validity with convergent thinking. This cross-linguistic flexibility allows for more inclusive, global-scale creativity research, addressing key limitations of earlier approaches. S-DAT provides a powerful tool for fairer, more comprehensive evaluation of cognitive flexibility in diverse populations and can be freely assessed online: https://sdat.iol.zib.de/.

Figures (5)

• Figure 1: Multilingual calibration results across different languages for the following models: text-embedding-3-large from OpenAI (row 1, left), nomic-embed-text-v1.5 (row 1, right), snowflake-arctic-embed-l (row 2, left), snowflake-arctic-embed-l-v2.0 (row 2, right), multilingual-e5-large-instruct (row 3, left), BGE-M3 (row 3, right), E5-Mistral-7B-instruct (row 4, left), and granite-embedding-278m-multilingual (row 4, right). While OpenAI's text-embedding-3-large model is a multilingual model, the pairwise dissimilarities of translated word pairs vary quite considerably across languages. The nomic-embed-text-v1.5 model is not a multilingual model, but rather trained for English language use, which leads to inflated pairwise dissimilarity values for non-English words written in the Latin script (used by languages such as English, German, Spanish, etc.) and for non-latin script (e.g., Russian, Chinese, Arabic, etc.) to significantly lower values and effectively collapses; the same holds for the snowflake-arctic-embed-l model. The snowflake-arctic-embed-l-v2.0 model is a multilingual model and shows a more stable calibration across languages; however also shows significant degradation for Chinese character-based languages like Japanese and Chinese, shown here for Japanese. While multilingual-e5-large-instruct and BGE-M3 are multilingual models, their pairwise dissimilarities are rather unstable across languages. E5-Mistral-7B-instruct has no stable calibration across languages. granite-embedding-278m-multilingual shows the best calibration across languages, including non-Latin script languages. Note that while granite-embedding-278m-multilingual has not been trained for Hindi, it still generates a distribution similar to the languages it has been trained for, which is most likely due to using the same tokenization as XLM-RoBERTa; whether this translates into proper semantic similarity scores will be investigated in future research.
• Figure 2: Multilingual calibration results across different languages for the granite-embedding-278m-multilingual model. Here the distributions across languages (including non-Latin script languages) are rather stable and comparable, which is important for the S-DAT.
• Figure 3: Correlation of embedding-based DAT scores with original DAT scores for the S-DAT (based on granite-embedding-278m-multilingual). The figure shows the relation between the calibrated S-DAT scores and the original DAT scores, illustrating the effectiveness of the calibration process and the alignment of the new model with the established DAT score, as well as the Alternative Use Task (AUT) and the Bridge-the-Associative-Gap Task (Bridge).
• Figure 4: Histograms displaying the distribution of the original DAT-scores as well as the S-DAT when applied to the data from Olson et al. (2021), Study 2. While the S-DAT's underlying granite-embedding-278m-multilingual-based semantic distances have been calibrated to match the mean and standard deviation of the DAT's underlying GloVe-based semantic distances, the S-DAT's score distribution (i.e., scoring the 10 provided pairs) has slightly lower standard deviation and a slightly higher mean than the DAT's score distribution. In particular, the S-DAT is slightly more robust to outliers; see also the transport graphic in Figure \ref{['fig:within']}.
• Figure 5: Comparison between the original DAT-scores as well as the S-DAT based on the data from olson_naming_2021, Study 2.