Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Multimodal Foundation AI and Expert-in-the-Loop for Sustainable Management of Wild Salmon Fisheries in Indigenous Rivers

Chi Xu, Yili Jin, Sami Ma, Rongsheng Qian, Hao Fang, Jiangchuan Liu, Xue Liu, Edith C. H. Ngai, William I. Atlas, Katrina M. Connors, Mark A. Spoljaric

TL;DR

This work tackles the challenge of sustainable wild salmon management in Indigenous rivers by integrating multimodal foundation AI with expert-in-the-loop frameworks. It combines video-based counting at weirs and sonar-based river monitoring, using cross-modal fusion and vision-language models to improve species identification, counting, and length estimation while maintaining ecological relevance through expert validation. The approach emphasizes edge-cloud deployment, continual and federated learning, and open data sharing to empower Indigenous communities and decision-makers with in-season, data-driven insights. By co-developing these tools with Indigenous stewards and conservation partners, the project aims to enhance management accuracy, support habitat restoration, and advance data sovereignty in a culturally informed, ethically governed AI ecosystem.

Abstract

Wild salmon are essential to the ecological, economic, and cultural sustainability of the North Pacific Rim. Yet climate variability, habitat loss, and data limitations in remote ecosystems that lack basic infrastructure support pose significant challenges to effective fisheries management. This project explores the integration of multimodal foundation AI and expert-in-the-loop frameworks to enhance wild salmon monitoring and sustainable fisheries management in Indigenous rivers across Pacific Northwest. By leveraging video and sonar-based monitoring, we develop AI-powered tools for automated species identification, counting, and length measurement, reducing manual effort, expediting delivery of results, and improving decision-making accuracy. Expert validation and active learning frameworks ensure ecological relevance while reducing annotation burdens. To address unique technical and societal challenges, we bring together a cross-domain, interdisciplinary team of university researchers, fisheries biologists, Indigenous stewardship practitioners, government agencies, and conservation organizations. Through these collaborations, our research fosters ethical AI co-development, open data sharing, and culturally informed fisheries management.

Exploring Multimodal Foundation AI and Expert-in-the-Loop for Sustainable Management of Wild Salmon Fisheries in Indigenous Rivers

TL;DR

This work tackles the challenge of sustainable wild salmon management in Indigenous rivers by integrating multimodal foundation AI with expert-in-the-loop frameworks. It combines video-based counting at weirs and sonar-based river monitoring, using cross-modal fusion and vision-language models to improve species identification, counting, and length estimation while maintaining ecological relevance through expert validation. The approach emphasizes edge-cloud deployment, continual and federated learning, and open data sharing to empower Indigenous communities and decision-makers with in-season, data-driven insights. By co-developing these tools with Indigenous stewards and conservation partners, the project aims to enhance management accuracy, support habitat restoration, and advance data sovereignty in a culturally informed, ethically governed AI ecosystem.

Abstract

Wild salmon are essential to the ecological, economic, and cultural sustainability of the North Pacific Rim. Yet climate variability, habitat loss, and data limitations in remote ecosystems that lack basic infrastructure support pose significant challenges to effective fisheries management. This project explores the integration of multimodal foundation AI and expert-in-the-loop frameworks to enhance wild salmon monitoring and sustainable fisheries management in Indigenous rivers across Pacific Northwest. By leveraging video and sonar-based monitoring, we develop AI-powered tools for automated species identification, counting, and length measurement, reducing manual effort, expediting delivery of results, and improving decision-making accuracy. Expert validation and active learning frameworks ensure ecological relevance while reducing annotation burdens. To address unique technical and societal challenges, we bring together a cross-domain, interdisciplinary team of university researchers, fisheries biologists, Indigenous stewardship practitioners, government agencies, and conservation organizations. Through these collaborations, our research fosters ethical AI co-development, open data sharing, and culturally informed fisheries management.
Paper Structure (15 sections, 8 figures)

This paper contains 15 sections, 8 figures.

Table of Contents

  1. Problem Statement
  2. Alignment with Sustainable Development Goals and the LNOB principle
  3. Strategies and Methods
  4. Multimodal Foundation AI for Salmon Monitoring
  5. Scalable and User-Centric Framework Design
  6. Collaboration and Open Data for Effective Fisheries Management
  7. Implementation Plan
  8. Expert-in-the-Loop AI for Salmon Monitoring
  9. Video-based Salmon Detection, Classification, and Counting
  10. Sonar-based Monitoring
  11. Evaluation Criteria
  12. Video-Based Salmon Species Identification and Counting
  13. Sonar-Based Salmon Monitoring
  14. Expected Results and Impacts
  15. Assumptions and Risks

Figures (8)

  • Figure 1: (a) A salmon counting weir at Koeye River (in Heiltsuk First Nation's traditional territory, northern British Columbia) with salmon swimming passing the fish channel, (b) sample underwater video frames with salmon appearances, (c) object segmentation with species identification.
  • Figure 2: (a) A mounted ARIS sonar camera, (b) sonar deployment in the Yakoun River, Haida Nation's traditional territory, (c) sample frames from ARIS sonar, (d) salmon detection and tracking in sonar frames.
  • Figure 3: An off-the-shelf vision language model (e.g., OpenAI o1) can identify some cases but also makes errors, thus requiring further refinement.
  • Figure 4: This project supports SalmonVision & Selective Fishery in multiple Indigenous rivers in British Columbia, Canada.
  • Figure 5: Vision language model verification and refinement.
  • ...and 3 more figures