LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System

TL;DR

This work tackles scalable autonomous port inspection by proposing a novel LLM-VLM fusion framework that combines LLM-driven symbolic planning for a cooperative UAV-USV team with VLM-based semantic inspection and reporting. It replaces traditional rule-based mission planners with a dependency-graph execution approach and leverages lightweight onboard VLMs to achieve real-time, context-aware perception. Validation in the MBZIRC Maritime Simulator and real-world trials demonstrates robust coordination, open-vocabulary understanding, and structured reporting on resource-constrained platforms. The framework lays a foundation for AI-driven maritime operations, enabling broader applications such as vessel monitoring, offshore infrastructure inspection, and environmental surveillance.

Abstract

Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large Language Models (LLMs) and Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection

Figures (21)

• Figure 1: LLM-VLM framework for autonomous maritime inspection. Mission instructions are processed by a LLM to generate symbolic plans and dependency graphs for UAV-USV coordination. The VLM then analyzes sensor data from the inspection scene to produce structured inspection reports, enabling semantic understanding, anomaly detection, and compliance assessment in dynamic port environments.
• Figure 2: Overview of the proposed LLM-VLM fusion framework for autonomous maritime inspection. Mission instructions and system prompts, including operational constraints and hazard knowledge, are processed by the LLM mission planner to generate symbolic plans and dependency graphs for safe UAV-USV coordination. The USV and UAV subsystems perform perception, waypoint guidance, and control, while the VLM Inspector provides semantic scene understanding, anomaly detection, and compliance assessment. The communication manager synchronizes information flow across platforms, and structured inspection reports are transmitted to the control center for operator review.
• Figure 3: Example of a system prompt used for planning port inspection missions with heterogeneous USV-UAV system. The prompt defines the system overview, operational environment, robot capabilities, coordination requirements, mission parameters, and expected response format. It guides the LLM to generate symbolic mission plans with spatial and sequential dependencies, enabling coordinated heterogeneous operations for maritime inspection tasks.
• Figure 4: Example of a symbolic mission plan for a heterogeneous USV-UAV system generated by the LLM. The plan specifies sequential actions with explicit dependencies between tasks to ensure safe and coordinated execution of maritime inspection operations.
• Figure 5: Map of the simulated port environment used for USV navigation and inspection tasks. The environment includes multiple docking stations, crane areas, container stacks, and open water regions, providing a realistic maritime setting to evaluate autonomous surface vehicle path planning and mission execution.