MERLION: Marine ExploRation with Language guIded Online iNformative Visual Sampling and Enhancement
Shrutika Vishal Thengane, Marcel Bartholomeus Prasetyo, Yu Xiang Tan, Malika Meghjani
TL;DR
MERLION addresses the challenge of selective, semantically relevant onboard video summarization for AUVs operating in murky waters by combining a CLIP-based presampling pipeline with an online sampling mechanism, and further enhances frames with a diffusion-based module in MERLION-E. The approach formalizes semantic relevance through a $512$-D CLIP embedding and a surprise-based sampling criterion, while MERLION-E introduces selective diffusion to improve frames before final sampling, maintaining real-time performance on edge hardware. Evaluation on three visibility conditions with a human-ground-truth SRUM benchmark shows that MERLION-E provides the best semantic and representative samples in low-visibility scenarios, and the framework remains practical for deployment on AUVs with modest compute budgets. The work contributes an open-source MERLION implementation and a standardized evaluation protocol, advancing semantically-guided underwater video summarization for real-time marine exploration.
Abstract
Autonomous and targeted underwater visual monitoring and exploration using Autonomous Underwater Vehicles (AUVs) can be a challenging task due to both online and offline constraints. The online constraints comprise limited onboard storage capacity and communication bandwidth to the surface, whereas the offline constraints entail the time and effort required for the selection of desired key frames from the video data. An example use case of targeted underwater visual monitoring is finding the most interesting visual frames of fish in a long sequence of an AUV's visual experience. This challenge of targeted informative sampling is further aggravated in murky waters with poor visibility. In this paper, we present MERLION, a novel framework that provides semantically aligned and visually enhanced summaries for murky underwater marine environment monitoring and exploration. Specifically, our framework integrates (a) an image-text model for semantically aligning the visual samples to the users' needs, (b) an image enhancement model for murky water visual data and (c) an informative sampler for summarizing the monitoring experience. We validate our proposed MERLION framework on real-world data with user studies and present qualitative and quantitative results using our evaluation metric and show improved results compared to the state-of-the-art approaches. We have open-sourced the code for MERLION at the following link https://github.com/MARVL-Lab/MERLION.git.