AquaVLM: Improving Underwater Situation Awareness with Mobile Vision Language Models

TL;DR

AquaVLM introduces a mobile vision-language model-based system to enable context-aware, underwater messaging via smartphones, addressing the limitations of predefined signals. The approach combines context-aware instruction tuning, hierarchical message generation, and error-resilient fine-tuning to maintain message integrity over acoustic channels. Evaluations in VR and on an iOS prototype demonstrate high purpose alignment, robust recovery under realistic BER, and practical resource usage suitable for consumer devices. The work advances underwater situational awareness by delivering informative, adaptable communication without bulky or expensive hardware, with potential applications in other extreme domains.

Abstract

Underwater activities like scuba diving enable millions annually to explore marine environments for recreation and scientific research. Maintaining situational awareness and effective communication are essential for diver safety. Traditional underwater communication systems are often bulky and expensive, limiting their accessibility to divers of all levels. While recent systems leverage lightweight smartphones and support text messaging, the messages are predefined and thus restrict context-specific communication. In this paper, we present AquaVLM, a tap-and-send underwater communication system that automatically generates context-aware messages and transmits them using ubiquitous smartphones. Our system features a mobile vision-language model (VLM) fine-tuned on an auto-generated underwater conversation dataset and employs a hierarchical message generation pipeline. We co-design the VLM and transmission, incorporating error-resilient fine-tuning to improve the system's robustness to transmission errors. We develop a VR simulator to enable users to experience AquaVLM in a realistic underwater environment and create a fully functional prototype on the iOS platform for real-world experiments. Both subjective and objective evaluations validate the effectiveness of AquaVLM and highlight its potential for personal underwater communication as well as broader mobile VLM applications.

Figures (10)

• Figure 1: Context-rich information generated and shared by $\sf{AquaVLM}$.
• Figure 2: System overview and workflow of $\sf{AquaVLM}$. In the offline stage (left), a customized VLM model is fine-tuned. In the online stage (right), the message is generated and transmitted between two mobile devices.
• Figure 3: $\sf{AquaVLM}$ user interface with an example where divers encounter a shark.
• Figure 4: Model instruction tuning pipeline.
• Figure 5: Error-resilient fine-tuning: Each bit of a character is randomly flipped given a BER.