Leveraging Multi-Modal Information to Enhance Dataset Distillation

TL;DR

<3-5 sentence high-level summary> This work tackles privacy-preserving dataset distillation by leveraging multi-modal information. It introduces a framework that uses automatically generated captions, bounding boxes, and segmentation masks to augment synthetic data through two caption-based strategies (caption concatenation and caption matching) and two object-centric losses (masked gradient matching and masked distribution matching). Extensive experiments on ImageNet-1K subsets show consistent improvements over state-of-the-art distillation methods and demonstrate strong cross-architecture generalization, especially when captions and masks are utilized. The results highlight the potential of multimodal supervision to produce more informative, privacy-aware synthetic datasets for downstream classification tasks.

Abstract

Dataset distillation aims to create a small and highly representative synthetic dataset that preserves the essential information of a larger real dataset. Beyond reducing storage and computational costs, related approaches offer a promising avenue for privacy preservation in computer vision by eliminating the need to store or share sensitive real-world images. Existing methods focus solely on optimizing visual representations, overlooking the potential of multi-modal information. In this work, we propose a multi-modal dataset distillation framework that incorporates two key enhancements: caption-guided supervision and object-centric masking. To leverage textual information, we introduce two strategies: caption concatenation, which fuses caption embeddings with visual features during classification, and caption matching, which enforces semantic alignment between real and synthetic data through a caption-based loss. To improve data utility and reduce unnecessary background noise, we employ segmentation masks to isolate target objects and introduce two novel losses: masked feature alignment and masked gradient matching, both aimed at promoting object-centric learning. Extensive evaluations demonstrate that our approach improves downstream performance while promoting privacy protection by minimizing exposure to real data.

Figures (5)

• Figure 1: Annotations of a sample from flamingo class.
• Figure 2: Overview of the Caption Combination Framework. (a) Caption Concatenation: The caption feature is integrated with the image feature before being passed through the linear layer for probability prediction. (b) Caption Matching: In each iteration, caption features are extracted from synthetic images and aligned with those from real images.
• Figure 3: Overview of the Mask Matching Framework.
• Figure 4: Qualitative results of different methods. (a) Macaw from ImNet-Birds, generated using caption concatenation. (b) The same Macaw class, generated using caption matching. (c), (d), and (e) show Parachute from ImNette, where mask-based methods effectively reduce background elements.
• Figure 5: Qualitative results at different resolutions. (a) Rapeseed from ImNet-A, with the left image at $128\times 128$ resolution and the right image at $256\times 256$. (b) Ruddy Turnstone from ImNet-B at the same setting.