The Impact of Responsible AI Research on Innovation and Development

TL;DR

This paper investigates the translational impact of Responsible AI (RAI) research on two real-world pathways: patents (innovation) and code repositories (development). It builds a large, multi-source dataset (200K papers spanning 1980–2022; 15M USPTO patents; 1M Papers With Code-linked repositories) and uses a Sentence-Transformers framework with 25 NIST AI RMF keywords to identify 1,747 RAI papers across top venues. The authors introduce four metrics to quantify impact, analyze time-to-impact via Kaplan–Meier survival analysis, and reveal a global, cross-disciplinary pattern where most translational impact stems from a small subset of papers, with repositories typically lagging patents by years. Key findings include a 1-year average to repository uptake and up to an 8-year lag to patent uptake, significant contributions from European and Asian institutions, and a tendency for RAI work to combine unconventional knowledge across domains. The work offers actionable recommendations for academia–industry collaboration, diversified and cross-disciplinary RAI research programs, and robust measurement practices to better map the translational landscape of responsible AI.

Abstract

Translational research, especially in the fast-evolving field of Artificial Intelligence (AI), is key to converting scientific findings into practical innovations. In Responsible AI (RAI) research, translational impact is often viewed through various pathways, including research papers, blogs, news articles, and the drafting of forthcoming AI legislation (e.g., the EU AI Act). However, the real-world impact of RAI research remains an underexplored area. Our study aims to capture it through two pathways: \emph{patents} and \emph{code repositories}, both of which provide a rich and structured source of data. Using a dataset of 200,000 papers from 1980 to 2022 in AI and related fields, including Computer Vision, Natural Language Processing, and Human-Computer Interaction, we developed a Sentence-Transformers Deep Learning framework to identify RAI papers. This framework calculates the semantic similarity between paper abstracts and a set of RAI keywords, which are derived from the NIST's AI Risk Management Framework; a framework that aims to enhance trustworthiness considerations in the design, development, use, and evaluation of AI products, services, and systems. We identified 1,747 RAI papers published in top venues such as CHI, CSCW, NeurIPS, FAccT, and AIES between 2015 and 2022. By analyzing these papers, we found that a small subset that goes into patents or repositories is highly cited, with the translational process taking between 1 year for repositories and up to 8 years for patents. Interestingly, impactful RAI research is not limited to top U.S. institutions, but significant contributions come from European and Asian institutions. Finally, the multidisciplinary nature of RAI papers, often incorporating knowledge from diverse fields of expertise, was evident as these papers tend to build on unconventional combinations of prior knowledge.

Figures (6)

• Figure 1: Data processing overview. a) curated papers from three sources, filtered them by year, and excluded non-research-oriented papers; b) identified RAI papers by comparing the embeddings (generated using ST5 ni2021sentence) of paper abstracts and selected NIST keywords; c) retrieved patent citations by matching the titles of patent references and those of RAI papers.
• Figure 2: Difference in academic citations of RAI papers going into patents and repositories. Each row represents a RAI topic, with the number of academic citations shown on a logarithmic scale. (a) Distributions of papers going into patents (blue) vs. those that do not (orange); and (b) distributions of papers going into repositories (blue) vs. those that do not (orange). Wider gap between the distributions in each plot indicates that papers going into patents tend to attract more academic citations that those that do not. Statistically significance difference between the means of the two distributions is at 0.001 level, and marked with a $*$.
• Figure 3: Number of papers for each RAI topic between 2015-2022. Since 2017, there has been an increasing trend of RAI papers, especially in Fairness and Privacy.
• Figure 4: Survival analysis of papers going into patents and repositories. For each topic, the probability of its papers going into: (a) patents for the first time (i.e., receiving the first citation), and (b) repositories for the first time (i.e., receiving the first star or fork on GitHub). The horizontal line at 0.5 represents the threshold beyond which papers have a better chance of going into patents or repositories than a coin flip.
• Figure 5: Conventionaliy of RAI papers going into patents and repositories. A topic's conventionality is based on whether its papers---those that going into patents (left plot) and those going into repositories (right plot)---have (un)usual combinations of citations. A larger red area indicates a higher degree of unconventionality. Papers going into patents in Accountability often cite papers discussing AI applications across different venues (e.g., KDD, ICLR, USENIX, CVPRW, NeurIPS).