Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications

Abby Stylianou, Michelle Brachman, Albatool Wazzan, Samuel Black, Richard Souvenir

TL;DR

This project designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user, and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected.

Abstract

The data that underlies automated methods in computer vision and machine learning, such as image retrieval and fine-grained recognition, often comes from crowdsourcing. In contexts that rely on the intrinsic motivation of users, we seek to understand how the application design affects a user's willingness to contribute and the quantity and quality of the data they capture. In this project, we designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected. The results suggest that higher levels of user labeling do not lead to reduced contribution. Users collected and annotated the most images using the application version with the highest requested level of labeling with no decrease in user satisfaction. In preliminary experiments, the additional labeled data supported increased performance on an image retrieval task.

Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications

TL;DR

This project designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user, and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected.

Abstract

The data that underlies automated methods in computer vision and machine learning, such as image retrieval and fine-grained recognition, often comes from crowdsourcing. In contexts that rely on the intrinsic motivation of users, we seek to understand how the application design affects a user's willingness to contribute and the quantity and quality of the data they capture. In this project, we designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected. The results suggest that higher levels of user labeling do not lead to reduced contribution. Users collected and annotated the most images using the application version with the highest requested level of labeling with no decrease in user satisfaction. In preliminary experiments, the additional labeled data supported increased performance on an image retrieval task.
Paper Structure (34 sections, 8 figures, 1 table)

This paper contains 34 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Application Design
  4. Unlabeled (UL)
  5. Weakly Labeled (WL)
  6. Strongly Labeled (SL)
  7. Experiment
  8. Study Protocol
  9. Participants
  10. Data
  11. User Interactions
  12. User Satisfaction Ratings
  13. Image Composition & Annotation Quality
  14. Results
  15. Number of Photos Taken
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Image retrieval systems are dependent on high quality, well labeled data. In this study, we designed and evaluated three variants of a camera-centric mobile application for crowdsourcing data for such a system. The variants require varying amounts of image labeling from the user: (from L to R) none, weak (naming objects), and strong (naming and locating objects).
  • Figure 2: Fieldguide, iNaturalist, and rePhoto are examples of camera-based mobile applications that require (a) no, (b) weak, and (c) strong labeling, respectively, from the users.
  • Figure 3: Design variants of the camera-based mobile application for capturing images and identifying objects in the scene.
  • Figure 4: (top) Number of images and (bottom) time in seconds spent capturing (and labeling) images using each application variant.
  • Figure 5: Example images captured using three design variants of the camera-based mobile application for capturing images of hotel rooms and identifying objects in the scene.
  • ...and 3 more figures