The Journey to Trustworthy AI: Pursuit of Pragmatic Frameworks

TL;DR

The paper argues that there is no universal framework for Trustworthy AI (TAI) due to contextual and normative variability across domains. It proposes a pragmatic, risk-based approach called Set-Formalize-Measure-Act (SFMA) to translate TAI attributes into concrete benchmarks, supporting governance and regulation without stifling innovation. It surveys the global regulatory landscape (US EU China), frames risk and uncertainty using the Rumsfeld Risk Matrix, and addresses fairness, explainability, and ongoing monitoring as core enablers of trustworthy deployment. A strong emphasis is placed on open-source collaboration and academia as engines of innovation, warning against over-regulation that could hinder TAI progress while advocating practical, measurable, and auditable frameworks. The work culminates in a multi-pronged path forward that leverages OSS, academic research, and modular frameworks to implement TAI responsibly across organizations and sectors.

Abstract

This paper reviews Trustworthy Artificial Intelligence (TAI) and its various definitions. Considering the principles respected in any society, TAI is often characterized by a few attributes, some of which have led to confusion in regulatory or engineering contexts. We argue against using terms such as Responsible or Ethical AI as substitutes for TAI. And to help clarify any confusion, we suggest leaving them behind. Given the subjectivity and complexity inherent in TAI, developing a universal framework is deemed infeasible. Instead, we advocate for approaches centered on addressing key attributes and properties such as fairness, bias, risk, security, explainability, and reliability. We examine the ongoing regulatory landscape, with a focus on initiatives in the EU, China, and the USA. We recognize that differences in AI regulations based on geopolitical and geographical reasons pose an additional challenge for multinational companies. We identify risk as a core factor in AI regulation and TAI. For example, as outlined in the EU-AI Act, organizations must gauge the risk level of their AI products to act accordingly (or risk hefty fines). We compare modalities of TAI implementation and how multiple cross-functional teams are engaged in the overall process. Thus, a brute force approach for enacting TAI renders its efficiency and agility, moot. To address this, we introduce our framework Set-Formalize-Measure-Act (SFMA). Our solution highlights the importance of transforming TAI-aware metrics, drivers of TAI, stakeholders, and business/legal requirements into actual benchmarks or tests. Finally, over-regulation driven by panic of powerful AI models can, in fact, harm TAI too. Based on GitHub user-activity data, in 2023, AI open-source projects rose to top projects by contributor account. Enabling innovation in TAI hinges on the independent contributions of the open-source community.

Figures (17)

• Figure 1: Main parties involved in assessing 'Trustworthy AI' in a product or service; Human end user (or community); Government; and the Private sector. Note that for every two entities, any acceptable TAI framework should be equipped to address the any professional (two-way) interactions.
• Figure 2: The first international summit on AI Safety held in November 2023 in Bletchley, UK. Twenty eight countries signed 'Bletchley Declaration'. List of countries retrieved from toney2023who.
• Figure 3: Timeline of data privacy laws passed by example legislative entities. While many countries have not yet passed or enacted their digital data privacy laws, pressured by public opinion, many legislative bodies will have completed their efforts to regulate AI within the next few years. For reference, in 1990, the first web sever and web browser were created by https://en.wikipedia.org/w/index.php?title=Tim_Berners-Lee&oldid=1210713982.
• Figure 4: A high-level understanding of EU, USA, and China's legal viewpoint towards regulating AI. Here, Stop, Caution, or Go are various responses to the important question: 'What should one do if existing laws may not have the capacity to regulate AI?'.
• Figure 5: Word cloud shown above created using President Biden's executive order (WHEO-14110) titled "The Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" (BidenAIEO2023). Larger words denote indicate that they are used more frequently in the text of WHEO-14110.