ORES-Inspect: A technology probe for machine learning audits on enwiki

TL;DR

The design of ORES-Inspect is described, an interface to enable editors to learn about and audit the performance of the ORES edit quality model and the plans for further research with this system.

Abstract

Auditing the machine learning (ML) models used on Wikipedia is important for ensuring that vandalism-detection processes remain fair and effective. However, conducting audits is challenging because stakeholders have diverse priorities and assembling evidence for a model's [in]efficacy is technically complex. We designed an interface to enable editors to learn about and audit the performance of the ORES edit quality model. ORES-Inspect is an open-source web tool and a provocative technology probe for researching how editors think about auditing the many ML models used on Wikipedia. We describe the design of ORES-Inspect and our plans for further research with this system.

Figures (1)

• Figure 1: The ORES-Inspect interface and login page, as accessible via Toolforge.