AssurAI: Experience with Constructing Korean Socio-cultural Datasets to Discover Potential Risks of Generative AI

TL;DR

This work addresses the shortage of non-English, culturally contextual AI safety benchmarks by introducing AssurAI, a quality-controlled Korean multimodal safety dataset. It defines a 35-risk-factor taxonomy drawn from international frameworks and Korea-specific contexts, and constructs 11,480 instances across text, image, video, and audio through a two-stage seed-and-crowdsourcing process with triple annotations and expert red-teaming. Pilot experiments across text and multimodal tracks demonstrate the dataset's capacity to reveal safety gaps and model refusal patterns, using standardized scoring and judge-model evaluation. The public release of AssurAI aims to advance safer and more reliable generative AI for the Korean community and informs broader, culturally aware AI safety benchmarking.

Abstract

The rapid evolution of generative AI necessitates robust safety evaluations. However, current safety datasets are predominantly English-centric, failing to capture specific risks in non-English, socio-cultural contexts such as Korean, and are often limited to the text modality. To address this gap, we introduce AssurAI, a new quality-controlled Korean multimodal dataset for evaluating the safety of generative AI. First, we define a taxonomy of 35 distinct AI risk factors, adapted from established frameworks by a multidisciplinary expert group to cover both universal harms and relevance to the Korean socio-cultural context. Second, leveraging this taxonomy, we construct and release AssurAI, a large-scale Korean multimodal dataset comprising 11,480 instances across text, image, video, and audio. Third, we apply the rigorous quality control process used to ensure data integrity, featuring a two-phase construction (i.e., expert-led seeding and crowdsourced scaling), triple independent annotation, and an iterative expert red-teaming loop. Our pilot study validates AssurAI's effectiveness in assessing the safety of recent LLMs. We release AssurAI to the public to facilitate the development of safer and more reliable generative AI systems for the Korean community.

Figures (8)

• Figure 1: The iterative construction and quality control process for the AssurAI dataset. Quality Assessment Team acts as the central hub, managing data quality through continuous feedback loops among Dataset Design Team, Data Production Team, and the Pilot Implementation Team.
• Figure 2: Distribution of 35 risk factors by total instances, stacked by data modality. The chart is sorted in descending order by the total number of instances. Modalities are color-coded: text (blue), image (orange), video (green), and audio (red).
• Figure 3: The overall safety evaluation pipeline of AssureAI, consisting of two tracks: the Text Track for text-based models and the Multimodal Track for audio, image, and video models.
• Figure 4: Result on the Safety Scores for Models
• Figure 5: Result on the Safety Scores for Prompt Types