MCM: Multi-layer Concept Map for Efficient Concept Learning from Masked Images

TL;DR

This work proposes Multi-layer Concept Map, the first work to devise an efficient concept learning method based on masked images, and introduces an asymmetric concept learning architecture by establishing correlations between different encoder and decoder layers, updating concept tokens using backward gradients from reconstruction tasks.

Abstract

Masking strategies commonly employed in natural language processing are still underexplored in vision tasks such as concept learning, where conventional methods typically rely on full images. However, using masked images diversifies perceptual inputs, potentially offering significant advantages in concept learning with large-scale Transformer models. To this end, we propose Multi-layer Concept Map (MCM), the first work to devise an efficient concept learning method based on masked images. In particular, we introduce an asymmetric concept learning architecture by establishing correlations between different encoder and decoder layers, updating concept tokens using backward gradients from reconstruction tasks. The learned concept tokens at various levels of granularity help either reconstruct the masked image patches by filling in gaps or guide the reconstruction results in a direction that reflects specific concepts. Moreover, we present both quantitative and qualitative results across a wide range of metrics, demonstrating that MCM significantly reduces computational costs by training on fewer than 75% of the total image patches while enhancing concept prediction performance. Additionally, editing specific concept tokens in the latent space enables targeted image generation from masked images, aligning both the visible contextual patches and the provided concepts. By further adjusting the testing time mask ratio, we could produce a range of reconstructions that blend the visible patches with the provided concepts, proportional to the chosen ratios.

Figures (7)

• Figure 1: (a) The architecture of the proposed Multi-layer Concept Map (MCM) method. (b) The detailed framework of the encoder and decoder layers.
• Figure 2: The t-SNE visualization of learned concept tokens in the latent space of MCM.
• Figure 3: Masked image reconstruction and editing results using a high test-time mask ratio of 75%.
• Figure 4: We could employ masks of any arbitrary size during the test phase. A larger mask size (e.g., 95%) provides a reconstruction that better represents the edited concepts, while a smaller mask size (e.g., 0%) generates images that align more closely with the contexts.
• Figure 5: Unbalanced concept classes in the CelebA dataset.