RiskRAG: A Data-Driven Solution for Improved AI Model Risk Reporting

TL;DR

RiskRAG addresses the gap in actionable AI risk reporting by leveraging a data-driven Retrieval-Augmented Generation pipeline that sources risks from a large corpus of model cards and AI incidents. The approach uses a five-design-requirement framework derived from literature and co-design with developers, and maps risks to real-world uses with prioritized mitigations. Empirical evaluation across baseline comparisons and multi-stakeholder user studies shows that RiskRAG provides more contextualized, structured, and actionable risk content, though it can reduce decision confidence as users become more risk-aware. The work has practical implications for risk documentation practices, platform integrations, and informed, responsible AI deployment and governance, with potential extensions to policy and public-facing risk communication.

Abstract

Risk reporting is essential for documenting AI models, yet only 14% of model cards mention risks, out of which 96% copying content from a small set of cards, leading to a lack of actionable insights. Existing proposals for improving model cards do not resolve these issues. To address this, we introduce RiskRAG, a Retrieval Augmented Generation based risk reporting solution guided by five design requirements we identified from literature, and co-design with 16 developers: identifying diverse model-specific risks, clearly presenting and prioritizing them, contextualizing for real-world uses, and offering actionable mitigation strategies. Drawing from 450K model cards and 600 real-world incidents, RiskRAG pre-populates contextualized risk reports. A preliminary study with 50 developers showed that they preferred RiskRAG over standard model cards, as it better met all the design requirements. A final study with 38 developers, 40 designers, and 37 media professionals showed that RiskRAG improved their way of selecting the AI model for a specific application, encouraging a more careful and deliberative decision-making. The RiskRAG project page is accessible at: https://social-dynamics.net/ai-risks/card.

Figures (8)

• Figure 1: Our approach consists of three steps. Step 1: We identified design requirements through literature and a co-design study with AI developers (§\ref{['sec:requirements']}). Each co-design session included an introduction ($\sim$5 min), a model card review ($\sim$15 min), and a co-design task ($\sim$20 min). After five iterations, we finalized the key design requirements. Step 2: We developed RiskRAG, using retrieval-augmented generation to generate risk reports aligned with these design requirements, leveraging data from model cards, and incident reports (§\ref{['sec:rag']}). Step 3: We evaluated RiskRAG reports in two user studies (§\ref{['sec:evaluation']}). In the preliminary study, $50$ AI developers compared a RiskRAG report to a baseline model card when assessing an AI model for a high-risk hiring scenario. In the final study, $38$ AI developers, $40$ UX designers, and $37$ media professionals compared RiskRAG reports to baseline model cards when selecting between two similar AI models for media industry tasks.
• Figure 2: Search query used for the literature review within the ACM DL repository.
• Figure 3: Architecture of RiskRAG. We denote with R1-R5 different steps aiming at fulfilling the design requirements from R1 to R5. The input to RiskRAG is a description of the model for which a risk report needs to be generated. The retriever first extracts risk-related content from the top-$k$ similar model cards and AI incidents (R1). The generator then adapts these risks into a standardized format and structures them using the risk taxonomy in weidingerSociotechnicalSafetyEvaluation2023(R2). The ExploreGen LLM module herdelExploreGenLargeLanguage2024 generates examples of real-world uses to which different risks are mapped to (R3). Mitigation strategies are similarly retrieved from model cards, formatted, and mapped to the corresponding risks (R4). Finally, risks are prioritized based on the number of uses they were mapped to and whether they have resulted in real-world incidents (R5).
• Figure 4: Baseline RiskRAG evaluation. This evaluation is performed on the evaluation set consisting of the top 10% most downloaded model cards (Table \ref{['tab:data_stats']}). We first produced risks R with RiskRAG for each of these cards using only their model descriptions. To assess the quality of RiskRAG's output against the existing risk sections of these cards, we parsed these risk sections through the generator (step 1) generating pseudo ground truth G to make them compatible with the risk content generated by RiskRAG, enabling direct comparison.
• Figure 5: Quantitative results from the preliminary user study: RiskRAG report outperformed baseline model cards across all the metrics. We had seven questions mapping to our design requirements to which participants were asked to answer on a Likert scale from 1 ("strongly disagree") to 5 ("strongly agree"). RiskRAG significantly outperformed the baseline model cards, with a one-point higher rating, moving from slight agreement to clear agreement on risk coverage, and from neutral to agreement on mitigation clarity.