Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models

TL;DR

The paper tackles how autonomous maritime systems can safely handle semantic hazards that go beyond geometric reasoning, in line with the currently drafted IMO MASS Code. It proposes Semantic Lookout, a camera-first, candidate-constrained vision-language model that selects a short-horizon, pre-approved fallback maneuver from water-safe trajectories during the alert-to-override window, with immediate operator override preserved. Through offline evaluation on 40 harbor scenes and a real sea trial, the approach demonstrates that fast VLMs can maintain substantial semantic awareness and that action choices align with human judgments better than geometry-only baselines, especially for clearly dangerous fire hazards where standoff increases are observed. The results motivate a pragmatic, domain-adapted hybrid autonomy paradigm that combines foundation-model semantics with conventional perception and planning, and highlight key HMI design considerations to support rapid, trustworthy operator handover. Future work aims to reduce latency with faster models and to extend sensing with bird’s-eye views and receding-horizon planning to broaden the safe action space beyond a single camera view.

Abstract

The draft IMO MASS Code requires autonomous and remotely supervised maritime vessels to detect departures from their operational design domain, enter a predefined fallback that notifies the operator, permit immediate human override, and avoid changing the voyage plan without approval. Meeting these obligations in the alert-to-takeover gap calls for a short-horizon, human-overridable fallback maneuver. Classical maritime autonomy stacks struggle when the correct action depends on meaning (e.g., diver-down flag means people in the water, fire close by means hazard). We argue (i) that vision-language models (VLMs) provide semantic awareness for such out-of-distribution situations, and (ii) that a fast-slow anomaly pipeline with a short-horizon, human-overridable fallback maneuver makes this practical in the handover window. We introduce Semantic Lookout, a camera-only, candidate-constrained vision-language model (VLM) fallback maneuver selector that selects one cautious action (or station-keeping) from water-valid, world-anchored trajectories under continuous human authority. On 40 harbor scenes we measure per-call scene understanding and latency, alignment with human consensus (model majority-of-three voting), short-horizon risk-relief on fire hazard scenes, and an on-water alert->fallback maneuver->operator handover. Sub-10 s models retain most of the awareness of slower state-of-the-art models. The fallback maneuver selector outperforms geometry-only baselines and increases standoff distance on fire scenes. A field run verifies end-to-end operation. These results support VLMs as semantic fallback maneuver selectors compatible with the draft IMO MASS Code, within practical latency budgets, and motivate future work on domain-adapted, hybrid autonomy that pairs foundation-model semantics with multi-sensor bird's-eye-view perception and short-horizon replanning.

Figures (21)

• Figure 1: System overview showing the three main modules: perception and proposal, decision and execution and authority.
• Figure 2: Coordinate frames and their relation. The water segmentation filters out any non-water obstacle in the field-of-view, in the projection $\Omega$.
• Figure 3: Block diagram for automatic waypoint follow with motion guidance on the ASV. A combined motion is the result of an override logic in which a human-in-the-loop may override the automatic motion guidance.
• Figure 4: Override switch logic implemented from autonomous operation to manual intervention and back. The percentage is the weight given to each actor, showing that during autonomy operations, the operator may intervene in the motion with both immediate action and with a gradual complete override by a defined phase-in time period.
• Figure 5: The Milliampere research ASV was used for data collection and in closed-loop experiments. Details in \ref{['app:platform']}.