An unsupervised learning approach to evaluate questionnaire data -- what one can learn from violations of measurement invariance

TL;DR

The paper tackles the problem of comparing questionnaire data across groups when measurement invariance cannot be assumed, a common issue in cross‑cultural research. It proposes an unsupervised three‑phase pipeline—data preparation (kNN imputation, group balancing, perturbation), Ward‑based clustering to identify typical response patterns (response types), and fingerprints that describe group composition by response type—to quantify inter‑group similarity without invariance requirements. By testing on synthetic datasets representing invariant, violated, and unrelated constructs, the authors demonstrate that their approach yields interpretable group descriptions and natural similarity measures, often outperforming PCA in the presence of non‑invariance. The method is descriptive but broadly applicable, robust to cluster count variations, and provides a practical toolkit for researchers needing cross‑group comparisons when traditional psychometric invariance checks are infeasible or insufficient.

Abstract

In several branches of the social sciences and humanities, surveys based on standardized questionnaires are a prominent research tool. While there are a variety of ways to analyze the data, some standard procedures have become established. When those surveys want to analyze differences in the answer patterns of different groups (e.g., countries, gender, age, ...), these procedures can only be carried out in a meaningful way if there is measurement invariance, i.e., the measured construct has psychometric equivalence across groups. As recently raised as an open problem by Sauerwein et al. (2021), new evaluation methods that work in the absence of measurement invariance are needed. This paper promotes an unsupervised learning-based approach to such research data by proposing a procedure that works in three phases: data preparation, clustering of questionnaires, and measuring similarity based on the obtained clustering and the properties of each group. We generate synthetic data in three data sets, which allows us to compare our approach with the PCA approach under measurement invariance and under violated measurement invariance. As a main result, we obtain that the approach provides a natural comparison between groups and a natural description of the response patterns of the groups. Moreover, it can be safely applied to a wide variety of data sets, even in the absence of measurement invariance. Finally, this approach allows us to translate (violations of) measurement invariance into a meaningful measure of similarity.

Figures (8)

• Figure 1: Graphical representation of the gap statistic as well as the dendrogram corresponding to the goodness of the clustering of the questionnaires in data set $\mathcal{D}_1$. Moreover, the corresponding response types are shown as a spider plot.
• Figure 2: The fingerprints of the different groups regarding the response types as spider plots. The radial $y$-axes are scaled to $(0, 0.7)$. Also, the group similarity on data set $\mathcal{D}_1$ is given by a dendrogram.
• Figure 3: Graphical representation of the gap statistic as well as the dendrogram corresponding to the goodness of the clustering of the questionnaires in data set $\mathcal{D}_2$. Moreover, the corresponding response types are shown as a spider plot.
• Figure 4: The fingerprints of the different groups regarding the response types as spider plots. The radial $y$-axes are scaled to $(0, 0.7)$. Moreover, the group similarity on data set $\mathcal{D}_2$ is shown as a dendrogram.
• Figure 5: Graphical representation of the gap statistic as well as the dendrogram corresponding to the goodness of the clustering of the questionnaires in data set $\mathcal{D}_3$. Moreover, the corresponding response types are shown as a spider plot.