Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios

Jiacheng Ruan, Wenzhen Yuan, Zehao Lin, Ning Liao, Zhiyu Li, Feiyu Xiong, Ting Liu, Yuzhuo Fu

TL;DR

The CamObj-Llava is proposed, an LVLM specifically designed for addressing tasks in camouflaged scenes, and a curriculum learning strategy with six distinct modes is introduced to facilitate the model's effective acquisition of knowledge about camouflaged objects and scenes.

Abstract

Large visual-language models (LVLMs) have achieved great success in multiple applications. However, they still encounter challenges in complex scenes, especially those involving camouflaged objects. This is primarily due to the lack of samples related to camouflaged scenes in the training dataset. To mitigate this issue, we construct the MM-CamObj dataset for the first time, comprising two subsets: CamObj-Align and CamObj-Instruct. Specifically, CamObj-Align contains 11,363 image-text pairs, and it is designed for VL alignment and injecting rich knowledge of camouflaged scenes into LVLMs. CamObj-Instruct is collected for fine-tuning the LVLMs with improved instruction-following capabilities, and it includes 11,363 images and 68,849 conversations with diverse instructions. Based on the MM-CamObj dataset, we propose the CamObj-Llava, an LVLM specifically designed for addressing tasks in camouflaged scenes. To facilitate our model's effective acquisition of knowledge about camouflaged objects and scenes, we introduce a curriculum learning strategy with six distinct modes. Additionally, we construct the CamObj-Bench to evaluate the existing LVLMs' capabilities of understanding, recognition, localization and count in camouflage scenes. This benchmark includes 600 images and 7 tasks, with a total of 9,449 questions. Extensive experiments are conducted on the CamObj-Bench with CamObj-Llava, 8 existing open-source and 3 closed-source LVLMs. Surprisingly, the results indicate that our model achieves a 25.84% improvement in 4 out of 7 tasks compared to GPT-4o. Code and datasets will be available at https://github.com/JCruan519/MM-CamObj.

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios

TL;DR

The CamObj-Llava is proposed, an LVLM specifically designed for addressing tasks in camouflaged scenes, and a curriculum learning strategy with six distinct modes is introduced to facilitate the model's effective acquisition of knowledge about camouflaged objects and scenes.

Abstract

Large visual-language models (LVLMs) have achieved great success in multiple applications. However, they still encounter challenges in complex scenes, especially those involving camouflaged objects. This is primarily due to the lack of samples related to camouflaged scenes in the training dataset. To mitigate this issue, we construct the MM-CamObj dataset for the first time, comprising two subsets: CamObj-Align and CamObj-Instruct. Specifically, CamObj-Align contains 11,363 image-text pairs, and it is designed for VL alignment and injecting rich knowledge of camouflaged scenes into LVLMs. CamObj-Instruct is collected for fine-tuning the LVLMs with improved instruction-following capabilities, and it includes 11,363 images and 68,849 conversations with diverse instructions. Based on the MM-CamObj dataset, we propose the CamObj-Llava, an LVLM specifically designed for addressing tasks in camouflaged scenes. To facilitate our model's effective acquisition of knowledge about camouflaged objects and scenes, we introduce a curriculum learning strategy with six distinct modes. Additionally, we construct the CamObj-Bench to evaluate the existing LVLMs' capabilities of understanding, recognition, localization and count in camouflage scenes. This benchmark includes 600 images and 7 tasks, with a total of 9,449 questions. Extensive experiments are conducted on the CamObj-Bench with CamObj-Llava, 8 existing open-source and 3 closed-source LVLMs. Surprisingly, the results indicate that our model achieves a 25.84% improvement in 4 out of 7 tasks compared to GPT-4o. Code and datasets will be available at https://github.com/JCruan519/MM-CamObj.
Paper Structure (22 sections, 5 figures, 6 tables)

This paper contains 22 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Camouflaged Scene Understanding (CSU) Datasets
  4. Large Vision and Language Models
  5. Datasets and Benchmark
  6. Metadata Construction
  7. CamObj-Align
  8. CamObj-Instruct
  9. CamObj-Bench
  10. Statistical Information
  11. Tasks Construction
  12. Methods
  13. Determination of Curriculum Learning Metrics
  14. Preliminary Experiment
  15. Experiments
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Empirical studies in camouflaged scenarios. We compare Llava-v1.5-7b and GPT-4o in understanding the two classical images with camouflage object existed. There is a certain gap between existing open-source models and advanced closed-source models in camouflage scenes.
  • Figure 2: An example prompt snippet for constructing CamObj-Align.
  • Figure 3: An example prompt snippet for constructing CamObj-Instruct.
  • Figure 4: Left: A sample example snippet in CamObj-Align. Right: A sample example snippet in CamObj-Instruct. The corresponding mask images are solely for better visualization of the camouflaged objects. These masks are not included in the CamObj-Align and CamObj-Instruct datasets and will not be used during the training process.
  • Figure 5: Some typical examples from the CamObj-Bench. Our benchmark consists of seven tasks to evaluate LVLMs' capabilities in recognition, classification, localization, counting, and scene understanding in the camouflaged object scenarios.