Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning

TL;DR

The paper addresses the risk of harm from machine-learning advances, especially synthetic media, and argues for nuanced release and publication practices rather than a binary open/closed stance. It proposes a framework grounded in risk assessment, external expert evaluation, and analogies from biosafety, information security, and IRBs to guide release decisions and governance. Key contributions include a hazard taxonomy (product, data, attention), an access-ratchet concept for progression to misuse, and a menu of release options with practical examples from industry. The work aims to reduce misuse while preserving openness by fostering community norms, expert input, and institutional support for responsible release practices.

Abstract

The aim of this paper is to facilitate nuanced discussion around research norms and practices to mitigate the harmful impacts of advances in machine learning (ML). We focus particularly on the use of ML to create "synthetic media" (e.g. to generate or manipulate audio, video, images, and text), and the question of what publication and release processes around such research might look like, though many of the considerations discussed will apply to ML research more broadly. We are not arguing for any specific approach on when or how research should be distributed, but instead try to lay out some useful tools, analogies, and options for thinking about these issues. We begin with some background on the idea that ML research might be misused in harmful ways, and why advances in synthetic media, in particular, are raising concerns. We then outline in more detail some of the different paths to harm from ML research, before reviewing research risk mitigation strategies in other fields and identifying components that seem most worth emulating in the ML and synthetic media research communities. Next, we outline some important dimensions of disagreement on these issues which risk polarizing conversations. Finally, we conclude with recommendations, suggesting that the machine learning community might benefit from: working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.