Position: Insights from Survey Methodology can Improve Training Data
Stephanie Eckman, Barbara Plank, Frauke Kreuter
TL;DR
This paper argues that AI labeling quality can be significantly improved by importing survey methodology, treating label collection as a data-collection task subject to cognitive and contextual biases. It connects label quality to a four-stage response model, outlines hypotheses about wording, format, order, and don’t know options, and proposes mitigation strategies such as randomization, instrument testing, paradata, and feedback. The authors advocate transparency and diversity in labeling processes and call for cross-disciplinary collaboration to ensure human-centric, fair, and accurate training data as AI systems scale. The practical impact is a concrete set of design and documentation practices to improve label accuracy and model alignment with public interests.
Abstract
Whether future AI models are fair, trustworthy, and aligned with the public's interests rests in part on our ability to collect accurate data about what we want the models to do. However, collecting high-quality data is difficult, and few AI/ML researchers are trained in data collection methods. Recent research in data-centric AI has show that higher quality training data leads to better performing models, making this the right moment to introduce AI/ML researchers to the field of survey methodology, the science of data collection. We summarize insights from the survey methodology literature and discuss how they can improve the quality of training and feedback data. We also suggest collaborative research ideas into how biases in data collection can be mitigated, making models more accurate and human-centric.