Enabling Fast and Accurate Crowdsourced Annotation for Elevation-Aware Flood Extent Mapping

TL;DR

FloodTrace tackles the need for fast, high-quality flood extent annotation to train elevation-aware models. It couples elevation-informed, web-based annotation tools with topological data analysis (persistent homology and contour-tree segmentation) and uncertainty-aware aggregation to enable effective crowdsourcing. In a study with 266 participants, elevation-guided tools improved labeling accuracy, while topology-based segmentation greatly reduced annotation time; corrected crowdsourced labels yielded models with performance close to those trained on expert-labeled data, dramatically cutting expert effort. The work demonstrates a scalable, end-to-end workflow for generating high-quality training data for flood extent mapping in disaster-management contexts.

Abstract

Mapping the extent of flood events is a necessary and important aspect of disaster management. In recent years, deep learning methods have evolved as an effective tool to quickly label high resolution imagery and provide necessary flood extent mappings. These methods, though, require large amounts of annotated training data to create models that are accurate and robust to new flooded imagery. In this work, we present FloodTrace, a web-based application that enables effective crowdsourcing of flooded region annotation for machine learning applications. To create this application, we conducted extensive interviews with domain experts to produce a set of formal requirements. Our work brings topological segmentation tools to the web and greatly improves annotation efficiency compared to the state-of-the-art. The user-friendliness of our solution allows researchers to outsource annotations to non-experts and utilize them to produce training data with equal quality to fully expert-labeled data. We conducted a user study to confirm the effectiveness of our application in which 266 graduate students annotated high-resolution aerial imagery from Hurricane Matthew in North Carolina. Experimental results show the efficiency benefits of our application for untrained users, with median annotation time less than half the state-of-the-art annotation method. In addition, using our aggregation and correction framework, flood detection models trained on crowdsourced annotations were able to achieve performance equal to models trained on fully expert-labeled annotations, while requiring a fraction of the time on the part of the expert.

Figures (7)

• Figure 1: An illustration of the full processing pipeline for FloodTrace. Input is given to the application in corresponding RGB imagery and elevation data. The elevation data is used in the server backend to create a 3D mesh which is combined with the RGB imagery as shown in \ref{['fig:mesh_creation']}. The elevation data is processed by the backend to create simplified topological segmentations at different levels of detail, as shown in \ref{['fig:segmentation_example']}. The user then utilizes the elevation-guided tools described by \ref{['sec:tools']} on the rendering in the web frontend. These annotations can then be downloaded for training machine learning models directly in the case of expert-labeled data, or for aggregation in the case of crowdsourcing. Aggregated annotations can be given as input to our application to be used with our uncertainty visualization tool for review and correction, as shown in \ref{['fig:aggregate']}, before being used to train machine learning models.
• Figure 2: An example showing mesh creation and rendering with aerial imagery RGB texture on a 1000 $\times$ 500 resolution example.
• Figure 3: An example region (A) with contour tree segmentations after no simplification (B) and at persistence thresholds $\epsilon$ = 0.02 (C), 0.04 (D), and 0.08 (E) of the elevation data function range. As seen in (B), segmentations produced without simplification are too noisy to be useful, while those in (E) correspond to data features such as the hills, lake, and river. Segmentations colored by rainbow colormap. Our application allows users to label these segmentations in one click to quickly annotate features as flooded or dry.
• Figure 4: Visualization of aggregated crowdsourced data from 45 participants. The aggregate view is shown in A) with no certainty threshold and in B) with a certainty threshold of 0.6. This view can quickly show the areas where the group of annotators were confident in labeling flooded or dry. The variance view is shown in C) with a threshold of 0.7 and compared in D) to a view with no annotation texture. With this view, it is easy to identify regions where there was high annotator disagreement, such as those shown in the orange borders. These selected regions are obviously flooded, and so the researcher can quickly correct these areas by labeling them. We explore how this can improve model performance in \ref{['sec:evaluation']}.
• Figure 5: UI for our application. Settings blown up for readability.