Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentation

TL;DR

This work tackles depth-constrained liver landmark segmentation by fusing RGB semantics and depth geometry through a dual-encoder pipeline (SAM2 for RGB and DA2 for depth). It introduces SRFT-GaLore, a scalable, gradient-based PEFT method that substitutes costly SVD with a structured random projection to efficiently fine-tune high-dimensional attention layers. A cross-attention fusion module integrates modalities, and a new LLSD external dataset enables robust cross-dataset evaluation. Empirical results on L3D show state-of-the-art DSC and reduced ASSD, with strong generalization to LLSD, illustrating practical potential for real-time, depth-guided intraoperative guidance. The approach enables scalable adaptation of foundation models to surgical domains, though future work aims to deepen cross-modal interactions and explore transformer-based decoders for enhanced global reasoning.

Abstract

Accurate detection and delineation of anatomical structures in medical imaging are critical for computer-assisted interventions, particularly in laparoscopic liver surgery where 2D video streams limit depth perception and complicate landmark localization. While recent works have leveraged monocular depth cues for enhanced landmark detection, challenges remain in fusing RGB and depth features and in efficiently adapting large-scale vision models to surgical domains. We propose a depth-guided liver landmark segmentation framework integrating semantic and geometric cues via vision foundation encoders. We employ Segment Anything Model V2 (SAM2) encoder to extract RGB features and Depth Anything V2 (DA2) encoder to extract depth-aware features. To efficiently adapt SAM2, we introduce SRFT-GaLore, a novel low-rank gradient projection method that replaces the computationally expensive SVD with a Subsampled Randomized Fourier Transform (SRFT). This enables efficient fine-tuning of high-dimensional attention layers without sacrificing representational power. A cross-attention fusion module further integrates RGB and depth cues. To assess cross-dataset generalization, we also construct a new Laparoscopic Liver Surgical Dataset (LLSD) as an external validation benchmark. On the public L3D dataset, our method achieves a 4.85% improvement in Dice Similarity Coefficient and a 11.78-point reduction in Average Symmetric Surface Distance compared to the D2GPLand. To further assess generalization capability, we evaluate our model on LLSD dataset. Our model maintains competitive performance and significantly outperforms SAM-based baselines, demonstrating strong cross-dataset robustness and adaptability to unseen surgical environments. These results demonstrate that our SRFT-GaLore-enhanced dual-encoder framework enables scalable and precise segmentation under real-time, depth-constrained surgical settings.

Figures (2)

• Figure 1: (a) Overview of our proposed model integrating SAM2 and DA2 encoders with cross-attention fusion module. Both encoders are fine-tuned using SRFT-GaLore for efficient adaptation. (b) Illustration of the SRFT-GaLore update, where the full gradient is projected through a SRFT matrix to obtain a compressed low-rank approximation $\tilde{G}_t \in \mathbb{R}^{128 \times 576}$.
• Figure 2: Segmentation performance of ours over SOTA in the classes of ridge, falciform ligament, and silhouette, with the color of red, blue and green.