Backdoor Attacks on Multi-modal Contrastive Learning

TL;DR

This work surveys backdoor and data-poisoning vulnerabilities in multi-modal contrastive learning, emphasizing that attacks targeting embedding geometry can transfer across downstream tasks with minimal poisoning. It details a taxonomy of attack stages, triggers, domains, and objectives, and reviews representative attacks in centralized, federated, graph, and multimodal settings. The paper also compares defense strategies, notably CleanCLIP and CleanerCLIP, highlighting their different focal points and complementary potential, while discussing open challenges such as certified defenses and industrial deployments. Overall, the findings underscore the need for representation-aware defenses and careful security considerations in self-supervised, distributed, and multi-modal systems.

Abstract

Contrastive learning has become a leading self- supervised approach to representation learning across domains, including vision, multimodal settings, graphs, and federated learning. However, recent studies have shown that contrastive learning is susceptible to backdoor and data poisoning attacks. In these attacks, adversaries can manipulate pretraining data or model updates to insert hidden malicious behavior. This paper offers a thorough and comparative review of backdoor attacks in contrastive learning. It analyzes threat models, attack methods, target domains, and available defenses. We summarize recent advancements in this area, underline the specific vulnerabilities inherent to contrastive learning, and discuss the challenges and future research directions. Our findings have significant implications for the secure deployment of systems in industrial and distributed environments.