Bridging Instead of Replacing Online Coding Communities with AI through Community-Enriched Chatbot Designs

TL;DR

The paper addresses the tendency of LLM-based chatbots to provide isolated, decontextualized assistance, which can erode participation in online coding communities. It introduces Community-Enriched AI, a paradigm that grounds AI responses in community-generated content and social cues from platforms like Kaggle, operationalized in the ChatCommunity system using a retrieval-augmented generation pipeline. Through two empirical studies (N=28 and N=12), it demonstrates that surface-level previews, author identities, and engagement signals can significantly boost trust, community engagement, and task performance, especially for higher-order data science reasoning. The work offers design guidelines for building AI assistants that augment rather than replace online communities, promoting socially informed learning and ongoing participation in peer-driven knowledge ecosystems.

Abstract

LLM-based chatbots like ChatGPT have become popular tools for assisting with coding tasks. However, they often produce isolated responses and lack mechanisms for social learning or contextual grounding. In contrast, online coding communities like Kaggle offer socially mediated learning environments that foster critical thinking, engagement, and a sense of belonging. Yet, growing reliance on LLMs risks diminishing participation in these communities and weakening their collaborative value. To address this, we propose Community-Enriched AI, a design paradigm that embeds social learning dynamics into LLM-based chatbots by surfacing user-generated content and social design feature from online coding communities. Using this paradigm, we implemented a RAG-based AI chatbot leveraging resources from Kaggle to validate our design. Across two empirical studies involving 28 and 12 data science learners, respectively, we found that Community-Enriched AI significantly enhances user trust, encourages engagement with community, and effectively supports learners in solving data science tasks. We conclude by discussing design implications for AI assistance systems that bridge -- rather than replace -- online coding communities.

Figures (4)

• Figure 1: Effects of ChatGPT over time (by week) on Stack Overflow question volumes per topic, with ChatGPT released on November 30th, 2022. Adapted from burtch2024.
• Figure 2: RAG Pipeline: The process begins with preprocessing the Kaggle notebooks from the Meta Kaggle Code dataset to extract non-empty, Python-written Jupyter notebooks, which are then organized by individual competitions. Each notebook is further divided into several chunks, and these chunks are linked with the corresponding metadata from the Meta Kaggle dataset through indexing. Next, the chunks are passed through an encoder to generate embeddings. The embeddings, along with the corresponding chunk content, are stored in ChromaDB. When a user inputs a query, the query is converted into embeddings, and our system searches for and ranks relevant chunks using user-selected sorting methods (relevance, votes, or views). The retrieved chunks, combined with the user’s query, are then formatted into a complete prompt and passed to LLMs to generate a coherent response. These responses are parsed by our system frontend using the react-markdown package and displayed as system outputs. The retrieved chunks, together with their metadata, are also shown in the source document panel below the system response.
• Figure 3: An overview of design variations: Design 1: Vanilla Link, Design 3: Community-Enriched Inline and Design 4: Community-Enriched Summary. Design 2: Community-Enriched Preview is the same as Figure \ref{['fig:fig1']}.Alpha.
• Figure 4: Perceived usefulness and reliability ranking: 1st represents the most useful/reliable, while 4th represents the least useful/reliable. The number on each bar indicates the number of participants who gave this ranking.