Revolutionizing Newcomers' Onboarding Process in OSS Communities: The Future AI Mentor

TL;DR

The paper tackles the challenge of onboarding OSS newcomers by proposing an AI mentor capable of guiding the entire process. It uses Design Fiction with 19 newcomers to derive 32 design strategies, builds an OSSerCopilot prototype integrated with GitHub, and validates it through interviews and TAM while conducting a literature review to locate research gaps. Key contributions include a comprehensive set of design strategies for an AI mentor across onboarding steps and evidence that newcomers find a GitHub-integrated AI mentor useful and acceptable, suggesting substantial practical impact for OSS sustainability. The work identifies critical gaps in current AI research—particularly for early onboarding stages like project discovery and structure comprehension—offering actionable directions for researchers and tool designers to advance newcomer-focused AI mentorship in OSS.

Abstract

Onboarding newcomers is vital for the sustainability of open-source software (OSS) projects. To lower barriers and increase engagement, OSS projects have dedicated experts who provide guidance for newcomers. However, timely responses are often hindered by experts' busy schedules. The recent rapid advancements of AI in software engineering have brought opportunities to leverage AI as a substitute for expert mentoring. However, the potential role of AI as a comprehensive mentor throughout the entire onboarding process remains unexplored. To identify design strategies of this ``AI mentor'', we applied Design Fiction as a participatory method with 19 OSS newcomers. We investigated their current onboarding experience and elicited 32 design strategies for future AI mentor. Participants envisioned AI mentor being integrated into OSS platforms like GitHub, where it could offer assistance to newcomers, such as ``recommending projects based on personalized requirements'' and ``assessing and categorizing project issues by difficulty''. We also collected participants' perceptions of a prototype, named ``OSSerCopilot'', that implemented the envisioned strategies. They found the interface useful and user-friendly, showing a willingness to use it in the future, which suggests the design strategies are effective. Finally, in order to identify the gaps between our design strategies and current research, we conducted a comprehensive literature review, evaluating the extent of existing research support for this concept. We find that research is relatively scarce in certain areas where newcomers highly anticipate AI mentor assistance, such as ``discovering an interested project''. Our study has the potential to revolutionize the current newcomer-expert mentorship and provides valuable insights for researchers and tool designers aiming to develop and enhance AI mentor systems.

Figures (9)

• Figure 1: Overview of the Research Design
• Figure 2: Difficulty Level of Different Steps of Newcomers' Onboarding Process
• Figure 3: Participants' Perceived Difficulty VS. Expectations for AI Mentor's Assistance
• Figure 4: Participants' Expectations for AI Mentor's Assistance VS. Satisfaction with Our Prototype
• Figure 5: Homepage of Prototype