AI Data Readiness Inspector (AIDRIN) for Quantitative Assessment of Data Readiness for AI

TL;DR

The paper addresses the lack of standardized methods to assess data readiness for AI and proposes AIDRIN, a comprehensive toolkit integrating traditional data quality metrics with AI-specific considerations. AIDRIN offers metrics across completeness, outliers, duplicates, privacy, feature correlations, fairness, FAIR compliance, class imbalance, and feature importance, plus visualizations and reports. Implemented in Python with a Flask web interface and a PyPI package, it demonstrates via SGC and TCGA-LUAD case studies, including a 2024 NCI CRDC AI Data-Readiness challenge, to showcase practical applicability. The evaluation covers performance, usability, and data-readiness outcomes, highlighting trade-offs such as SHAP-based feature importance being time-intensive, and outlining a path toward a unified AIDRIN score for standardized readiness assessment and future expansion to multi-modal data and federated settings.

Abstract

"Garbage In Garbage Out" is a universally agreed quote by computer scientists from various domains, including Artificial Intelligence (AI). As data is the fuel for AI, models trained on low-quality, biased data are often ineffective. Computer scientists who use AI invest a considerable amount of time and effort in preparing the data for AI. However, there are no standard methods or frameworks for assessing the "readiness" of data for AI. To provide a quantifiable assessment of the readiness of data for AI processes, we define parameters of AI data readiness and introduce AIDRIN (AI Data Readiness Inspector). AIDRIN is a framework covering a broad range of readiness dimensions available in the literature that aid in evaluating the readiness of data quantitatively and qualitatively. AIDRIN uses metrics in traditional data quality assessment such as completeness, outliers, and duplicates for data evaluation. Furthermore, AIDRIN uses metrics specific to assess data for AI, such as feature importance, feature correlations, class imbalance, fairness, privacy, and FAIR (Findability, Accessibility, Interoperability, and Reusability) principle compliance. AIDRIN provides visualizations and reports to assist data scientists in further investigating the readiness of data. The AIDRIN framework enhances the efficiency of the machine learning pipeline to make informed decisions on data readiness for AI applications.

Figures (9)

• Figure 1: An overview of the AIDRIN workflow. The AI readiness assessment starts by processing data and metadata to generate summary statistics. Users can then select metrics across different data readiness dimensions for evaluation. AIDRIN generates corresponding visualizations and a comprehensive report of the selected metrics, aiding in data readiness analysis and decision-making.
• Figure 2: Interface for uploading files and summarizing dataset statistics in AIDRIN. Users can import datasets and view key statistical summaries, the dimensions of the dataset, and the numerical and categorical features.
• Figure 3: AIDRIN enables a tailored evaluation of the data and models through customized analysis.
• Figure 4: AIDRIN user interface for selecting metrics for the German Credit Datasets hofmann1994statlog dataset.
• Figure 5: A sample of the visualizations generated by AIDRIN based on user-selected metrics during SGC dataset evaluation.