Where are the Whales: A Human-in-the-loop Detection Method for Identifying Whales in High-resolution Satellite Imagery

TL;DR

The paper addresses scalable whale monitoring in open oceans by reframing detection as a statistically grounded anomaly-detection problem on very-high-resolution satellite imagery. It introduces a human-in-the-loop pipeline: (i) compute a deviation tensor via two standardization strategies, (ii) aggregate to an anomaly map and extract interesting points, and (iii) review these points through a web-based labeling interface. The approach achieves high recalls (≈$90.3\%$–$96.4\%$) while dramatically reducing expert inspection area (up to $99.8\%$), demonstrated on three annotated scenes, and is open-sourced to enable reproducibility and community adoption. While not fully automated, the method provides a scalable first step toward marine mammal monitoring from space, bootstrapping larger ML pipelines with expert-validated labels and enabling rapid, large-scale data collection for conservation action.

Abstract

Effective monitoring of whale populations is critical for conservation, but traditional survey methods are expensive and difficult to scale. While prior work has shown that whales can be identified in very high-resolution (VHR) satellite imagery, large-scale automated detection remains challenging due to a lack of annotated imagery, variability in image quality and environmental conditions, and the cost of building robust machine learning pipelines over massive remote sensing archives. We present a semi-automated approach for surfacing possible whale detections in VHR imagery using a statistical anomaly detection method that flags spatial outliers, i.e. "interesting points". We pair this detector with a web-based labeling interface designed to enable experts to quickly annotate the interesting points. We evaluate our system on three benchmark scenes with known whale annotations and achieve recalls of 90.3% to 96.4%, while reducing the area requiring expert inspection by up to 99.8% -- from over 1,000 sq km to less than 2 sq km in some cases. Our method does not rely on labeled training data and offers a scalable first step toward future machine-assisted marine mammal monitoring from space. We have open sourced this pipeline at https://github.com/microsoft/whales.

Figures (6)

• Figure 1: Web-based labeling application for quickly annotating "interesting points". For each candidate region of imagery flagged as statistically anomalous, labelers can inspect the corresponding imagery (the scale of each pixel is overlaid on the image), assign confidence levels (Possible, Probable, Definite), and classify object type (e.g., whale, ship, debris, zooplankton). Metadata such as location, date, and chips ID are displayed alongside adjustable image rendering controls (zoom, brightness, contrast). Once a class button is pressed the interface automatically loads a new image/interesting point to minimize the time per annotation.
• Figure 2: Comparison of numerical error in variance estimation under 32-bit precision for the naive and mean-shifted formulations of computing standard deviation in the rolling window approach. The $x$-axis shows the ratio of variance to mean intensity for synthetic image chips. The $y$-axis shows the absolute error in standard deviation relative to a double-precision ground truth. The naive formulation suffers from catastrophic cancellation when variance is small compared to the mean (which happens when using the rolling window approach over homogeneous ocean regions), while the shifted method achieves consistently lower error across all regimes.
• Figure 3: (Top) The distribution of anomaly values (summed absolute deviations) on the Cape Cod Bay scene hodul2022individual using the two standardization methods with varying kernel ($k$) or window ($s$) sizes. We find that both methods result in similar distributions with a window or kernel size greater than 50. (Bottom) Comparison of anomaly value distributions (using rolling window standardization with a $k=51$ kernel) across the three evaluation scenes. The Valdés 2014 scene has a high density of high-contrast features (mostly white-capped waves) which shows up as a longer tail in the anomaly distribution.
• Figure 4: Example whale detections from the May 22, 2020 scene (left) and two detections from the larger Cape Cod Bay 2021 scene that were not considered in the Hodul et al. 2022 study (middle, right). The May 2020 scene (1,056 sq km) produced 220 interesting points, each of which were labeled by 3 experts in a total of 31 minutes. The Cape Cod Bay scene (1,083 sq km) produced 555 interesting points, each was also labeled by 3 experts in a total of 139 minutes.
• Figure 5: A map of 555 "interesting points" in red found over the two WorldView-3 scenes from used in the hodul2022individual study. The whale points from the Hodul et al. study are shown in green and the study area (buffered from the extent of the labeled points) is shown in yellow. Each of the "interesting points" was annotated by three expert labelers which took a total of 139 minutes. Manual annotation of the entire scene would take approximately $\sim 35$ hours. The interesting points covered 28 out of the 31 whales found in the Hodul et al. study (which were all correctly identified by the labelers). Further, 5 additional whale points were found outside of the previously examined study area.