Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Co-ML: Collaborative Machine Learning Model Building for Developing Dataset Design Practices

Tiffany Tseng, Matt J. Davidson, Luis Morales-Navarro, Jennifer King Chen, Victoria Delaney, Mark Leibowitz, Jazbo Beason, R. Benjamin Shapiro

TL;DR

Co-ML introduces a collaborative ML modeling platform designed to teach dataset design practices to youth, addressing the data-centric gap in novice ML education. Through a 2-week AIML Summer Camp, teams used Co-ML to collaboratively collect data, train image classifiers, test performance, and iterate on datasets, guided by four dataset design practices: diversity, performance evaluation, balancing, and data quality. The study shows that collaboration and testing interfaces enable richer engagement with data issues, leading to more diverse datasets, insights into model performance, and increased learner confidence in dataset design. The work highlights the potential of distributed data collection and hands-on, learner-driven projects to promote authentic ML understanding and responsible data practices for future technologists.

Abstract

Machine learning (ML) models are fundamentally shaped by data, and building inclusive ML systems requires significant considerations around how to design representative datasets. Yet, few novice-oriented ML modeling tools are designed to foster hands-on learning of dataset design practices, including how to design for data diversity and inspect for data quality. To this end, we outline a set of four data design practices (DDPs) for designing inclusive ML models and share how we designed a tablet-based application called Co-ML to foster learning of DDPs through a collaborative ML model building experience. With Co-ML, beginners can build image classifiers through a distributed experience where data is synchronized across multiple devices, enabling multiple users to iteratively refine ML datasets in discussion and coordination with their peers. We deployed Co-ML in a 2-week-long educational AIML Summer Camp, where youth ages 13-18 worked in groups to build custom ML-powered mobile applications. Our analysis reveals how multi-user model building with Co-ML, in the context of student-driven projects created during the summer camp, supported development of DDPs including incorporating data diversity, evaluating model performance, and inspecting for data quality. Additionally, we found that students' attempts to improve model performance often prioritized learnability over class balance. Through this work, we highlight how the combination of collaboration, model testing interfaces, and student-driven projects can empower learners to actively engage in exploring the role of data in ML systems.

Co-ML: Collaborative Machine Learning Model Building for Developing Dataset Design Practices

TL;DR

Co-ML introduces a collaborative ML modeling platform designed to teach dataset design practices to youth, addressing the data-centric gap in novice ML education. Through a 2-week AIML Summer Camp, teams used Co-ML to collaboratively collect data, train image classifiers, test performance, and iterate on datasets, guided by four dataset design practices: diversity, performance evaluation, balancing, and data quality. The study shows that collaboration and testing interfaces enable richer engagement with data issues, leading to more diverse datasets, insights into model performance, and increased learner confidence in dataset design. The work highlights the potential of distributed data collection and hands-on, learner-driven projects to promote authentic ML understanding and responsible data practices for future technologists.

Abstract

Machine learning (ML) models are fundamentally shaped by data, and building inclusive ML systems requires significant considerations around how to design representative datasets. Yet, few novice-oriented ML modeling tools are designed to foster hands-on learning of dataset design practices, including how to design for data diversity and inspect for data quality. To this end, we outline a set of four data design practices (DDPs) for designing inclusive ML models and share how we designed a tablet-based application called Co-ML to foster learning of DDPs through a collaborative ML model building experience. With Co-ML, beginners can build image classifiers through a distributed experience where data is synchronized across multiple devices, enabling multiple users to iteratively refine ML datasets in discussion and coordination with their peers. We deployed Co-ML in a 2-week-long educational AIML Summer Camp, where youth ages 13-18 worked in groups to build custom ML-powered mobile applications. Our analysis reveals how multi-user model building with Co-ML, in the context of student-driven projects created during the summer camp, supported development of DDPs including incorporating data diversity, evaluating model performance, and inspecting for data quality. Additionally, we found that students' attempts to improve model performance often prioritized learnability over class balance. Through this work, we highlight how the combination of collaboration, model testing interfaces, and student-driven projects can empower learners to actively engage in exploring the role of data in ML systems.
Paper Structure (46 sections, 12 figures, 4 tables)

This paper contains 46 sections, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Role of Data in ML
  4. ML Education for Youth
  5. Collaborative Learning
  6. Research Opportunities for Fostering Dataset Design Practices for Novices
  7. Dataset Design Practices
  8. Co-ML System and Companion Starter App
  9. Co-ML Design Goals
  10. Co-ML System Description
  11. Define ontology of labels in a classifier
  12. Collect Data
  13. Train a Model
  14. Evaluate Model Performance
  15. Iterating on a model
  16. ...and 31 more sections

Figures (12)

  • Figure 1: Data collection interface for adding labelled images to the shared dataset (left). All images added across devices are visible in the synchronized Training Data Dashboard, organized by label name (right).
  • Figure 2: Classifying new data using the Photo Classification Mode (left) and Live Classification Mode (right). In Photo Classification Mode, the user takes a photograph and can review the classification results. For misclassified data, they can relabel it with the correct label. In Live Classification, users can see an updating bar chart displaying relative confidence levels for each class in their model.
  • Figure 3: Testing mode interfaces. Users can review collectively added test data, and classification results are based on the latest trained local model. Tapping on a misclassified sample shows a bar chart of confidence levels to help users debug or improve model performance.
  • Figure 4: Model Evaluation Game. Users fulfill as many rounds as they can within a 90 second time limit. Each round is 5 seconds, and the user gets a target object to show to the camera. The round score is calculated based on the confidence level of the image classified at the end of each round.
  • Figure 5: Starter app for building custom ML-powered app using models built in Co-ML. The app includes a customizable launch screen, where the purpose of the app and items the app can classify can be described. The app then launches into a camera interface, where the information displayed when an item is classified can be customized. The content that appears in the camera view is defined by a JSON file; in this example, when the model returns a classification result of "tomato," the app displays a number ("water") for the gallons of water needed to grow a tomato and a corresponding tomato emoji ("emoji").
  • ...and 7 more figures