Learning Collaborative Knowledge with Multimodal Representation for Polyp Re-Identification

TL;DR

This work tackles colonoscopic polyp ReID by addressing domain gaps and the limits of unimodal representations through a visual-text multimodal approach. It introduces DMCL, which pairs a ResNet-50 image encoder with a BERT text encoder and fuses their features via a self-attention based dynamic collaborative learning module for end-to-end training. The training objective combines $L_{Triplet}$ and $L_{ID}$ to form $L_{total}$, with careful triplet sampling to learn discriminative multimodal embeddings. Empirical results on Colo-Pair and standard ReID benchmarks show that DMCL with dynamic multimodal fusion achieves state-of-the-art performance, demonstrating the practical value of multimodal collaboration in clinical polyp recognition and retrieval.

Abstract

Colonoscopic Polyp Re-Identification aims to match the same polyp from a large gallery with images from different views taken using different cameras, which plays an important role in the prevention and treatment of colorectal cancer in computer-aided diagnosis. However, traditional methods for object ReID directly adopting CNN models trained on the ImageNet dataset usually produce unsatisfactory retrieval performance on colonoscopic datasets due to the large domain gap. Worsely, these solutions typically learn unimodal modal representations on the basis of visual samples, which fails to explore complementary information from other different modalities. To address this challenge, we propose a novel Deep Multimodal Collaborative Learning framework named DMCL for polyp re-identification, which can effectively encourage multimodal knowledge collaboration and reinforce generalization capability in medical scenarios. On the basis of it, a dynamic multimodal feature fusion strategy is introduced to leverage the optimized visual-text representations for multimodal fusion via end-to-end training. Experiments on the standard benchmarks show the benefits of the multimodal setting over state-of-the-art unimodal ReID models, especially when combined with the collaborative multimodal fusion strategy. The code is publicly available at https://github.com/JeremyXSC/DMCL.

Figures (6)

• Figure 1: The task diagram of colonoscopic polyp re-identification, which matches and correlates polyps appearing across different temporal frames, viewing angles, or clinical examinations. The core technical challenge involves determining whether polyps visualized in distinct images or video sequences represent the same pathological lesion.
• Figure 2: The illustration of our multimodal polyp dataset with visual image and its corresponding text description. Specially, given a query image, the main goal of this work is to learn a robust polyp re-identification model on the basis of visual-text representation.
• Figure 3: The overview of our proposed method on polyp re-identification task, which contains two main parts: 1) visual feature backbone and 2) textual feature backbone respectively. Specially, visual feature backbone is composed of CNN network, and textual feature backbone is consisted of Transformer architecture named Bert model. In addition, a multimodal fusion strategy is introduced to mine the mutual benefits between visual feature and texture feature, which can further boost the performance of proposed DMCL framework.
• Figure 4: The t-SNE visualization of our method with different pre-training settings (e.g. (a) DMCL w/o Text and (b) DMCL) on Colo-Pair dataset. Note that points of the same color represent the same class.
• Figure 5: Qualitative visualization of ranking results of our proposed approach DMCL with collaborative training mechanism on Colo-Pair dataset. For each group, the first column shows the query image, while the second and third columns display the inferred results retrieved by our DMCL model. What's more, we can obviously observe that our method shows a great robustness regardness of the deformation or illumination variation of these captured polyps.