Exploring Learners' Expectations and Engagement When Collaborating with Constructively Controversial Peer Agents

TL;DR

This study investigates how constructive controversy CC can be instantiated in LLM-based peer agents for asynchronous learning. Using a mixed-method 2x2 design, it contrasts regulated versus unregulated CC behaviors and the impact of disclosing the agent design on learner engagement, sense of agency, and learning outcomes. It identifies two learner orientations, Efficiency-Driven and Curiosity-Driven, that interact with agent behavior to shape collaboration strategies; findings show regulated CC increases discourse time and pushback without improving argument quality, while unregulated CC better supports emotional engagement for some learners. The work provides design implications for tailoring CC-based AI peers to learner profiles and highlights the tradeoffs of transparency in agent behavior, offering foundational guidance for integrating peer AI in isolated educational settings.

Abstract

Peer agents can supplement real-time collaborative learning in asynchronous online courses. Constructive Controversy (CC) theory suggests that humans deepen their understanding of a topic by confronting and resolving controversies. This study explores whether CC's benefits apply to LLM-based peer agents, focusing on the impact of agents' disputatious behaviors and disclosure of agents' behavior designs on the learning process. In our mixed-method study (n=144), we compare LLMs that follow detailed CC guidelines (regulated) to those guided by broader goals (unregulated) and examine the effects of disclosing the agents' design to users (transparent vs. opaque). Findings show that learners' values influence their agent interaction: those valuing control appreciate unregulated agents' willingness to cease push-back upon request, while those valuing intellectual challenges favor regulated agents for stimulating creativity. Additionally, design transparency lowers learners' perception of agents' abilities. Our findings lay the foundation for designing effective collaborative peer agents in isolated educational settings.

Figures (12)

• Figure 1: A summary diagram of the study procedure. After watching the experiment demo video, each participant completes two rounds of the task for the two behavior mechanisms (regulated and unregulated) with a 5-minute break in between. The dashed-border box shows the steps within each round. Note that, for both rounds, each participant experiences the same level of the peer agent's behavior design disclosure. The grey box shows the step for only participants in the transparent group (skipped by participants in the opaque group.)
• Figure 2: This figure shows the experiment interface (a). The screen is divided into three columns. The leftmost column is the activity instruction, including the task scenario, the activity steps, and the argument grading rubrics. The middle column is the chat window, where the participant has a conversation with the peer agent. The rightmost column is a template for the final argument write-up. At the beginning of the activity, the popup window provides the debate prompt (b), for which the participant picks their team's stance. After the participant finishes the task, they annotate the peer agent's messages (c). By hovering over the square next to each message, they can mark the message as Too Cooperative or Too Contradictory compared to what they expected from an AI peer and/or a human peer. Demo video of the experimental interface can be found in the supplementary materials.
• Figure 3: A schema showing the prompting pipeline of the peer agents. The boxes represent separate LLM threads, and the text on each arrow refers to the information getting passed between the threads. From "start", the black arrows (solid and dashed) show the peer agent initialization process, while the green arrows show actions during the collaboration process for behavior moderation.
• Figure 4: Examples of conversations between (a) a EDL participant and the regulated agent, (b) a EDL participant and the unregulated agent, (c) an CDL participant and the regulated agent, and (d) an CDL participant and the unregulated agent.
• Figure 5: Box plots showing the participants' observed measures of engagement from the task and their argument scores. The three plots in the upper rows are for turn counts, task completion time, and average word count per turn, respectively. All three plots are of data aggregated from all 72 participants. The two plots in the lower row are on argument strength (out of 10) and argument variance (out of 10), respectively. The X-axes are based on the learners' orientations. The colors of all five box plots indicate the peer agent's behavior mechanisms. The statistically significant comparisons are marked with asterisks (*: $p<.05$, **: $p<.01$, ***:$p<.001$).