Provocation on Expertise in Social Impact Evaluations of Generative AI (and Beyond)

TL;DR

This provocation begins to think carefully about the types of experts and expertise that are needed to conduct robust social impact evaluations of generative AI, and suggests that doing so will require thoughtfully eliciting and integrating insights from a range of domain experts and experiential experts.

Abstract

Social impact evaluations are emerging as a useful tool to understand, document, and evaluate the societal impacts of generative AI. In this provocation, we begin to think carefully about the types of experts and expertise that are needed to conduct robust social impact evaluations of generative AI. We suggest that doing so will require thoughtfully eliciting and integrating insights from a range of "domain experts" and "experiential experts," and close with five open questions.