Modeling Collaborative Problem Solving Dynamics from Group Discourse: A Text-Mining Approach with Synergy Degree Model

TL;DR

This study addresses the challenge of measuring collaborative problem solving (CPS) synergy by integrating automated discourse analysis with the Synergy Degree Model (SDM) to quantify group dynamics in a cMOOC. It conducts a large-scale, AI-assisted labeling of CPS behaviors across four interaction levels, comparing nine models (including BERT and GPT variants) and demonstrating that GPT offers high precision suitable for human-in-the-loop coding, while BERT achieves solid overall accuracy. Applying SDM, the authors derive weekly order parameters for each CPS subsystem and a composite synergy degree, showing that automated measures largely preserve construct validity and reveal meaningful patterns: task type influences lower-level order parameters, while collaborative quality strongly differentiates overall synergy. These findings support the feasibility of AI-in-the-loop CPS analytics for scalable formative assessment and provide actionable insights for designing collaborative learning environments. The work contributes a rigorous pipeline from discourse to system dynamics, highlights the value of synergy degree as a sensitive predictor of collaborative quality, and outlines limitations and future directions for multi-level modeling and cross-context validation.

Abstract

Measuring collaborative problem solving (CPS) synergy remains challenging in learning analytics, as classical manual coding cannot capture emergent system-level dynamics. This study introduces a computational framework that integrates automated discourse analysis with the Synergy Degree Model (SDM) to quantify CPS synergy from group communication. Data were collected from 52 learners in 12 groups during a 5-week connectivist MOOC (cMOOC) activity. Nine classification models were applied to automatically identify ten CPS behaviors across four interaction levels: operation, wayfinding, sense-making, and creation. While BERT achieved the highest accuracy, GPT models demonstrated superior precision suitable for human-AI collaborative coding. Within the SDM framework, each interaction level was treated as a subsystem to compute group-level order parameters and derive synergy degrees. Permutation tests showed automated measures preserve construct validity, despite systematic biases at the subsystem level. Statistical analyses revealed significant task-type differences: survey study groups exhibited higher creation-order than mode study groups, suggesting "controlled disorder" may benefit complex problem solving. Importantly, synergy degree distinguished collaborative quality, ranging from excellent to failing groups. Findings establish synergy degree as a sensitive indicator of collaboration and demonstrate the feasibility of scaling fine-grained CPS analytics through AI-in-the-loop approaches.

Figures (2)

• Figure 1: The research design and procedure
• Figure 2: The Permutation Test of CPS Order Parameter and Synergy between Human vs. Model