Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radar

Runwei Guan, Jianan Liu, Liye Jia, Haocheng Zhao, Shanliang Yao, Xiaohui Zhu, Ka Lok Man, Eng Gee Lim, Jeremy Smith, Yutao Yue

TL;DR

This work tackles the need for low-power, on-device visual grounding for USVs by integrating camera and 4D radar inputs under natural-language prompts. It introduces NanoMVG, a one-stage, multi-task model that performs both REC and RES with a lightweight Triplet-Modal Dynamic Fusion (TMDF) and an Edge-Neighbour Mixture-of-Expert (EN-MoE) to efficiently fuse modalities and allocate features for detection and segmentation. Training uses a joint loss with uncertainty weighting and specialized REC/RES heads, achieving state-of-the-art balance between accuracy and power on WaterVG, while maintaining real-time inference on embedded hardware. The approach demonstrates strong generalization to standard grounding benchmarks and offers practical impact for persistent, privacy-conscious waterway monitoring on USVs.

Abstract

Recently, visual grounding and multi-sensors setting have been incorporated into perception system for terrestrial autonomous driving systems and Unmanned Surface Vehicles (USVs), yet the high complexity of modern learning-based visual grounding model using multi-sensors prevents such model to be deployed on USVs in the real-life. To this end, we design a low-power multi-task model named NanoMVG for waterway embodied perception, guiding both camera and 4D millimeter-wave radar to locate specific object(s) through natural language. NanoMVG can perform both box-level and mask-level visual grounding tasks simultaneously. Compared to other visual grounding models, NanoMVG achieves highly competitive performance on the WaterVG dataset, particularly in harsh environments and boasts ultra-low power consumption for long endurance.

NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radar

TL;DR

This work tackles the need for low-power, on-device visual grounding for USVs by integrating camera and 4D radar inputs under natural-language prompts. It introduces NanoMVG, a one-stage, multi-task model that performs both REC and RES with a lightweight Triplet-Modal Dynamic Fusion (TMDF) and an Edge-Neighbour Mixture-of-Expert (EN-MoE) to efficiently fuse modalities and allocate features for detection and segmentation. Training uses a joint loss with uncertainty weighting and specialized REC/RES heads, achieving state-of-the-art balance between accuracy and power on WaterVG, while maintaining real-time inference on embedded hardware. The approach demonstrates strong generalization to standard grounding benchmarks and offers practical impact for persistent, privacy-conscious waterway monitoring on USVs.

Abstract

Recently, visual grounding and multi-sensors setting have been incorporated into perception system for terrestrial autonomous driving systems and Unmanned Surface Vehicles (USVs), yet the high complexity of modern learning-based visual grounding model using multi-sensors prevents such model to be deployed on USVs in the real-life. To this end, we design a low-power multi-task model named NanoMVG for waterway embodied perception, guiding both camera and 4D millimeter-wave radar to locate specific object(s) through natural language. NanoMVG can perform both box-level and mask-level visual grounding tasks simultaneously. Compared to other visual grounding models, NanoMVG achieves highly competitive performance on the WaterVG dataset, particularly in harsh environments and boasts ultra-low power consumption for long endurance.
Paper Structure (18 sections, 11 equations, 6 figures, 5 tables)

This paper contains 18 sections, 11 equations, 6 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Works
  3. Waterway Perception based on Camera-Radar Fusion
  4. Visual Grounding Models
  5. Method
  6. Overview of Pipeline
  7. Triplet-Modal Dynamic Fusion
  8. Edge-Neighour Mixture-of-Expert
  9. Prediction Heads of REC and RES
  10. Training Objectives
  11. Experiments
  12. Settings of Experiments
  13. Overall Comparison with State-of-the-Arts
  14. Comparison on Power Consumption and Inference Speed
  15. Ablation Experiments on Key Components
  16. ...and 3 more sections

Figures (6)

  • Figure 1: The pipeline of our proposed NanoMVG.
  • Figure 2: The architecture of NanoMVG. FoV indicates Front of View. The Triplet-Modal Dynamic Fusion (TMDF) and EN-MoE are two our proposed core modules.
  • Figure 3: Edge-Neighbour Mixture-of-Expert (EN-MoE).
  • Figure 4: The structure of REC and RES heads.
  • Figure 5: The prediction of NanoMVG-S. The prediction includes the predicted bounding boxes (REC) and segmented masks (RES).
  • ...and 1 more figures