Parameter-Efficient Fine-Tuning of Vision Foundation Model for Forest Floor Segmentation from UAV Imagery

TL;DR

This work adapts the Segment Anything Model (SAM), a vision foundation model with strong generalization capabilities, to segment forest floor objects such as tree stumps, vegetation, and woody debris, and employs parameter-efficient fine-tuning (PEFT) to fine-tune a small subset of additional model parameters while keeping the original weights fixed.

Abstract

Unmanned Aerial Vehicles (UAVs) are increasingly used for reforestation and forest monitoring, including seed dispersal in hard-to-reach terrains. However, a detailed understanding of the forest floor remains a challenge due to high natural variability, quickly changing environmental parameters, and ambiguous annotations due to unclear definitions. To address this issue, we adapt the Segment Anything Model (SAM), a vision foundation model with strong generalization capabilities, to segment forest floor objects such as tree stumps, vegetation, and woody debris. To this end, we employ parameter-efficient fine-tuning (PEFT) to fine-tune a small subset of additional model parameters while keeping the original weights fixed. We adjust SAM's mask decoder to generate masks corresponding to our dataset categories, allowing for automatic segmentation without manual prompting. Our results show that the adapter-based PEFT method achieves the highest mean intersection over union (mIoU), while Low-rank Adaptation (LoRA), with fewer parameters, offers a lightweight alternative for resource-constrained UAV platforms.

Figures (11)

• Figure 1: Adapter injects two serial adapters into each transformer block houlsby2019parameter. One adapter is injected after the Multi-Head Attention module, and the other is injected after the MLP module.
• Figure 2: AdaptFormer introduces a bottleneck adapter alongside with a MLP layer named AdaptMLP chen2022adaptformer.
• Figure 3: LoRA architecture with a trainable matrix product BA hu2022lora, where the original weight matrix remains frozen. The final embedding $h$ is obtained by element-wise summation of the outputs from the pre-trained weights and the LoRA module.
• Figure 4: Modified Segment Anything architecture with PEFT modules. The image encoder is frozen and the PEFT modules and the whole mask decoder parameters are trainable.
• Figure 5: Example of the orthomosaic image from field-D (left), and its corresponding mask labels (right), featuring coarse woody debris (CWD), miscellaneous (MISC), tree stumps (STUMP), and vegetation.