Comparing Native and Non-native English Speakers' Behaviors in Collaborative Writing through Visual Analytics

TL;DR

COALA addresses how NS and NNS behave differently in collaborative writing by combining ensemble sequence clustering, multi-granularity visualizations, and LLM-generated summaries to support interpretable analysis. The method uses a dataset of 162 sessions from 27 teams and validates the tool through focused and individual user studies with domain experts and researchers. Key contributions include uncertainty visualization across clustering, two complementary clustering/summarization methods, and a practical framework for AI-assisted collaborative writing tools. The work demonstrates actionable insights into multilingual collaboration and offers design guidelines for extending analysis to other collaborative processes.

Abstract

Understanding collaborative writing dynamics between native speakers (NS) and non-native speakers (NNS) is critical for enhancing collaboration quality and team inclusivity. In this paper, we partnered with communication researchers to develop visual analytics solutions for comparing NS and NNS behaviors in 162 writing sessions across 27 teams. The primary challenges in analyzing writing behaviors are data complexity and the uncertainties introduced by automated methods. In response, we present \textsc{COALA}, a novel visual analytics tool that improves model interpretability by displaying uncertainties in author clusters, generating behavior summaries using large language models, and visualizing writing-related actions at multiple granularities. We validated the effectiveness of \textsc{COALA} through user studies with domain experts (N=2+2) and researchers with relevant experience (N=8). We present the insights discovered by participants using \textsc{COALA}, suggest features for future AI-assisted collaborative writing tools, and discuss the broader implications for analyzing collaborative processes beyond writing.

Figures (9)

• Figure 1: Turn-taking of NS (native speaker) and NNS (non-native speaker) of English in the multilingual collaborative writing study.
• Figure 2: All sequences of non-native speakers' (NNS) and native speakers's (NS) behaviors during the individual and collaborative writing stage. The sequences in the collaborative writing stage are longer due to the concatenation of turns. The length of the rectangles indicates the duration of each event, and the color encodes the event types.
• Figure 3: Six patterns returned by Sequence Synopsis chen2017sequence for native speakers during the collaborative writing stage. Each pattern is a sequence of circles, representing a visual overview of sequences belonging to the cluster. The original sequences of the currently selected pattern are displayed below. They are sequences of rectangles, with event duration encoded by length. Events matched to the pattern are outlined in black.
• Figure 4: Consensus of clusters: sequences assigned to the same cluster by both Sequence Synopsis and hierarchical clustering are in the box; other sequences are outside of the box. Users can also change the number of clusters by dragging the slider, or manually add new clusters.
• Figure 5: Design variants: tree, transition matrix, and arc diagram (final design) to visualize sequence information for author NNS-15.