AI LEGO: Scaffolding Cross-Functional Collaboration in Industrial Responsible AI Practices during Early Design Stages

TL;DR

AI LEGO tackles the knowledge handoff barrier in cross-functional industrial Responsible AI by providing stage-based Lifecycle Blocks, Stage-centered Evaluation prompts, and LLM-driven Persona-centered Evaluation to surface harms early. A formative co-design with eight practitioners informs three design goals, which are instantiated in an interactive web tool evaluated against a Google Docs baseline with 18 practitioners across AI, PM, and UX roles. Results show AI LEGO Full increases the number of identified harms by ~195% and yields higher perceived usefulness and integration, while Eight-Stage prompts and persona simulations support structured, multi-perspective harm evaluation. The study demonstrates practical gains for proactive harm anticipation in early AI design and outlines design opportunities and limitations for integrating such scaffolds into real industrial workflows.

Abstract

Responsible AI (RAI) efforts increasingly emphasize the importance of addressing potential harms early in the AI development lifecycle through social-technical lenses. However, in cross-functional industry teams, this work is often stalled by a persistent knowledge handoff challenge: the difficulty of transferring high-level, early-stage technical design rationales from technical experts to non-technical or user-facing roles for ethical evaluation and harm identification. Through literature review and a co-design study with 8 practitioners, we unpack how this challenge manifests -- technical design choices are rarely handed off in ways that support meaningful engagement by non-technical roles; collaborative workflows lack shared, visual structures to support mutual understanding; and non-technical practitioners are left without scaffolds for systematic harm evaluation. Existing tools like JIRA or Google Docs, while useful for product tracking, are ill-suited for supporting joint harm identification across roles, often requiring significant extra effort to align understanding. To address this, we developed AI LEGO, a web-based prototype that supports cross-functional AI practitioners in effectively facilitating knowledge handoff and identifying harmful design choices in the early design stages. Technical roles use interactive blocks to draft development plans, while non-technical roles engage with those blocks through stage-specific checklists and LLM-driven persona simulations to surface potential harms. In a study with 18 cross-functional practitioners, AI LEGO increased the volume and likelihood of harms identified compared to baseline worksheets. Participants found that its modular structure and persona prompts made harm identification more accessible, fostering clearer and more collaborative RAI practices in early design.

Figures (5)

• Figure 1: Two-phase team-based evaluation study task flow.
• Figure 2: Effect of conditions on (A) Count, Severity, and Likelihood of identified harms and (B) Usability and Preference. AI LEGO Full allowed AI teams to identify more potential harms with a higher likelihood; it was more preferred and considered well-integrated, especially in the planning phase. Significance levels: *** $p < .001$, ** $p < .01$, * $p < .05$. Error bars indicate 95% CI.
• Figure 3: Concepts & Artifacts explored in the co-design study. Planning: (A) Plain description presents currently unstructured approaches to articulating AI development plans in cross-functional teams; (B) Stage-based scaffolding breaks down and connects AI development stages while providing prompts, as inspired by AI storyboard cramer2019challenges. Harm identification: (C) The survey was adapted from Value Cards shen2021value investigating different social-technical dimensions, (D) Stakeholder table visualizes AI development stages versus stakeholders for better sensemaking processes, (E) Commenting represents common features in asynchronous communication tools.
• Figure 4: Key components and transitions in AI LEGO user interface.
• Figure 5: Scenario & Condition orders in the user study.