Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing the validity of multiple opinion dynamics models

Samuel Moor-Smith, Dino Carpentras

TL;DR

This study tackles the validation of opinion-dynamics models by adopting a data-science-like calibration-validation workflow that splits data into training and prediction waves and quantifies predictive performance with a Wasserstein-based prediction error and an explained-variance metric relative to a null model. Four models (two toy: Deffuant, HK; two empirical: ED, Duggins) are calibrated via hyperparameter optimization, with experiments on synthetic ground-truth data and real-world European Social Survey (ESS) data. Across synthetic data, models can reproduce generator dynamics and sometimes outperform the null; however, on ESS data all models converge to freezing behavior with explained variance around zero, indicating incompatibility with real-world dynamics under the tested frameworks. The results underscore the need for more expressive features and modeling assumptions to capture the temporal evolution of opinions in real populations and to establish robust benchmarks for model validity and policy-relevant applications.

Abstract

While opinion dynamics models have been extensively studied as stylized models, there has been growing attention to the possibility of combining these models with empirical data. This attention seems to be driven by the many social issues that strongly depend on people's opinions (such as climate change and vaccination) and the need for empirically valid models to design related policy interventions. While different models have been combined in various ways with empirical data, standardised comparison of models against empirical data is still lacking. In this article, we test the validity of multiple opinion dynamics models--including both stylized and more realistic models. Our approach follows a "data science-like" validation procedure, where we first calibrate the model's free parameters using an initial range of years (e.g. 2010-2015), and then use data from one wave (e.g. 2016) to predict data in the following wave (e.g. 2017). We initially tested such a procedure using simulated data and then tested different models on various topics from the European Social Survey. Both toy models and empirical models perform well on the simulated data, but fail to predict future years in the empirical data. Furthermore, during the calibration phase on the empirical data, most models learned to "freeze"--meaning that their predictions for the following year are just a copy of the data from the previous year. This work advances the literature by offering a benchmark for comparing different opinion dynamics models. Furthermore, our tests show that real-world dynamics appear to be completely incompatible with the dynamics of the tested models. This calls for more effort in exploring what are the features that would improve validity and applications for opinion dynamics models.

Testing the validity of multiple opinion dynamics models

TL;DR

This study tackles the validation of opinion-dynamics models by adopting a data-science-like calibration-validation workflow that splits data into training and prediction waves and quantifies predictive performance with a Wasserstein-based prediction error and an explained-variance metric relative to a null model. Four models (two toy: Deffuant, HK; two empirical: ED, Duggins) are calibrated via hyperparameter optimization, with experiments on synthetic ground-truth data and real-world European Social Survey (ESS) data. Across synthetic data, models can reproduce generator dynamics and sometimes outperform the null; however, on ESS data all models converge to freezing behavior with explained variance around zero, indicating incompatibility with real-world dynamics under the tested frameworks. The results underscore the need for more expressive features and modeling assumptions to capture the temporal evolution of opinions in real populations and to establish robust benchmarks for model validity and policy-relevant applications.

Abstract

While opinion dynamics models have been extensively studied as stylized models, there has been growing attention to the possibility of combining these models with empirical data. This attention seems to be driven by the many social issues that strongly depend on people's opinions (such as climate change and vaccination) and the need for empirically valid models to design related policy interventions. While different models have been combined in various ways with empirical data, standardised comparison of models against empirical data is still lacking. In this article, we test the validity of multiple opinion dynamics models--including both stylized and more realistic models. Our approach follows a "data science-like" validation procedure, where we first calibrate the model's free parameters using an initial range of years (e.g. 2010-2015), and then use data from one wave (e.g. 2016) to predict data in the following wave (e.g. 2017). We initially tested such a procedure using simulated data and then tested different models on various topics from the European Social Survey. Both toy models and empirical models perform well on the simulated data, but fail to predict future years in the empirical data. Furthermore, during the calibration phase on the empirical data, most models learned to "freeze"--meaning that their predictions for the following year are just a copy of the data from the previous year. This work advances the literature by offering a benchmark for comparing different opinion dynamics models. Furthermore, our tests show that real-world dynamics appear to be completely incompatible with the dynamics of the tested models. This calls for more effort in exploring what are the features that would improve validity and applications for opinion dynamics models.
Paper Structure (23 sections, 7 equations, 3 figures)

This paper contains 23 sections, 7 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Methodology for Model Calibration and Validation
  3. Terminology and notation
  4. Datasets:
  5. Models as operators:
  6. Prediction Error:
  7. Explained Variance and Opinion Drift:
  8. Models and Free Parameters
  9. Optimization Method
  10. Simulated Data: Experiments 1 - 3
  11. Generation of the (Simulated) Ground-Truth Data.
  12. Experiment 1: Model's Reproducibility
  13. Opinion Change Required for Model Reconstruction.
  14. A Note on the Duggins Model and Reproducibility
  15. Results of experiment 1
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Explained variance of the optimized model (coloured) and generator model (grey) as a function of $\Delta_{\text{op}}(D)$. For the (grey) ground-truth case (refer. Sec. \ref{['reproducibility']}), deterministic models like Deffuant and HK require little opinion drift to outperform the null baseline, while more stochastic models like ED and Duggins require substantially more. The Duggins model shows high variance in repeated runs, reflecting its limited reproducibility. Furthermore, The optimizer matches ground-truth performance at high $\Delta_{\text{op}}(D)$ and often exceeds it at low $\Delta_{\text{op}}(D)$ by mimicking the null model. Shaded areas indicate standard deviation across runs.
  • Figure 2: (left) The optimizer’s performance on a noisy version of a ground-truth dataset. While performance declines exponentially with noise for the Deffuant, HK, and ED models, the Duggins model remains low and variable even without added noise. (right) Visualization of optimized values fit to the ground-truth data when the ground-truth data is generated from the Deffuant model.
  • Figure 3: Strip plot showing explained variance for the four models across six ESS datasets. Each strip reflects the distribution of explained variance over multiple runs. The black diamonds and bars are the mean and standard deviation of the runs. Across datasets, models rarely outperform the null model, with explained variance clustering around zero—suggesting the optimizer is learning to "freeze."