"Shall We Dig Deeper?": Designing and Evaluating Strategies for LLM Agents to Advance Knowledge Co-Construction in Asynchronous Online Discussions

TL;DR

This paper tackles the stagnation of knowledge co-construction in asynchronous discussions by designing a process-aware AI agent that advances discussion phases through phase-specific interventions. Grounded in IAM and related models, a design-workshop-derived set of task-oriented and relationship-oriented strategies guides an LLM-based agent that monitors phase sufficiency and applies targeted prompts. In a within-subject study (N=60) across five consecutive threads, the agent consistently promoted deeper progression relative to a human-only baseline, with Telling, Selling, and Participating yielding distinct benefits for initiation, exploration, and negotiation, while Delegating showed minimal impact. The findings demonstrate the viability of adaptive, phase-sensitive AI support to sustain more constructive online discourse, and they offer design implications for balancing task goals with relational engagement in multi-party AI-assisted collaboration.

Abstract

Asynchronous online discussions enable diverse participants to co-construct knowledge beyond individual contributions. This process ideally evolves through sequential phases, from superficial information exchange to deeper synthesis. However, many discussions stagnate in the early stages. Existing AI interventions typically target isolated phases, lacking mechanisms to progressively advance knowledge co-construction, and the impacts of different intervention styles in this context remain unclear and warrant investigation. To address these gaps, we conducted a design workshop to explore AI intervention strategies (task-oriented and/or relationship-oriented) throughout the knowledge co-construction process, and implemented them in an LLM-powered agent capable of facilitating progression while consolidating foundations at each phase. A within-subject study (N=60) involving five consecutive asynchronous discussions showed that the agent consistently promoted deeper knowledge progression, with different styles exerting distinct effects on both content and experience. These findings provide actionable guidance for designing adaptive AI agents that sustain more constructive online discussions.

Figures (12)

• Figure 1: The screenshot shows a Stack Exchange discussion thread responding to the question “What language for starting on Linux?”, where the accepted answer (top) recommends learning pseudocode, C, and any scripting language except C++. Comments A, B, and C co-construct knowledge by complementing, elaborating, or challenging this suggestion. Comment A agrees with and expands on the value of pseudocode. Comment B reinforces the recommendation while adding specific language examples such as Python and PHP. In contrast, Comment C challenges the practicality of learning pseudocode for beginners and raises concerns about its lack of standard syntax. Together, these interactions illustrate how community members collaboratively refine and contest knowledge claims, contributing to a richer collective understanding.
• Figure 2: Activities of the workshop. Sufficiency criteria for each phase were defined by the full cohort (N = 12), while representative intervention strategies for each agent style were first synthesized in subgroups (N = 4) and then validated by all participants (N = 12)
• Figure 3: Architecture of the LLM agent implemented in our experiment, detailing its key components and operational logic. The architecture integrates high-level intervention logic to support targeted interventions in multi-party discussions, thereby facilitating the progressive advancement of knowledge co-construction.
• Figure 4: Example screenshots from four threads on the discussion platform used in the lab study, illustrating the interface and instances of agent interventions across different phases and styles. Participants engaged in discussions by posting comments and liking or replying to others. The examples show: (1) a delegating-style intervention tailored to Phase 1, (2) a telling-style intervention tailored to Phase 2, (3) a participating-style intervention tailored to Phase 3, and (4) a selling-style intervention tailored to Phase 4.
• Figure 5: The counterbalancing strategy employed in the study. We iteratively sampled initial sequences and applied the template permutation, discarding duplicates until we obtained six seeds that yielded 60 unique condition sequences.