SmartCS: Enabling the Creation of ML-Powered Computer Vision Mobile Apps for Citizen Science Applications without Coding

TL;DR

SmartCS tackles the barrier of building ML-powered citizen-science mobile apps by providing a no-code platform that enables on-device, offline computer vision guidance. Its three-step workflow—dataset creation, on-device model training, and mobile app construction—uses lightweight mobile-optimized models and TensorFlow Lite to deliver real-time CV in the field. The authors validate the platform with six use cases (six apps) and two user studies, showing that non-programmers can create apps and end-users benefit from ML-guided data collection, even without internet access. This work lowers development costs and accelerates deployment of CV-enabled citizen-science apps, potentially broadening participation and improving data quality and educational value. It also identifies avenues for improvement, including more templates, richer UI, and human-ML collaboration for better accuracy and learning feedback.

Abstract

It is undeniable that citizen science contributes to the advancement of various fields of study. There are now software tools that facilitate the development of citizen science apps. However, apps developed with these tools rely on individual human skills to correctly collect useful data. Machine learning (ML)-aided apps provide on-field guidance to citizen scientists on data collection tasks. However, these apps rely on server-side ML support, and therefore need a reliable internet connection. Furthermore, the development of citizen science apps with ML support requires a significant investment of time and money. For some projects, this barrier may preclude the use of citizen science effectively. We present a platform that democratizes citizen science by making it accessible to a much broader audience of both researchers and participants. The SmartCS platform allows one to create citizen science apps with ML support quickly and without coding skills. Apps developed using SmartCS have client-side ML support, making them usable in the field, even when there is no internet connection. The client-side ML helps educate users to better recognize the subjects, thereby enabling high-quality data collection. We present several citizen science apps created using SmartCS, some of which were conceived and created by high school students.

Figures (10)

• Figure 1: Overview and workflow of the components of our open-source citizen science app creation platform.
• Figure 2: Server-side (left) vs client-side (right) machine learning (ML) models. We used client-side ML models in our implementation, which provided real-time object detection without any network connectivity or server-side processing requirements.
• Figure 3: The web version of the platform was created for easy access and uses different feasible computational resources for different steps.
• Figure 4: This figure illustrates the type of materials that the “Recycle This” app can detect papers, aluminum cans, plastic containers, and glass bottles.
• Figure 5: Appearance of the RipSnap app with examples of rip currents detected by the app. The location of the rip current is visualized using the red bounding box with the label and the confidence score of detection.