Bridging the gap to real-world language-grounded visual concept learning

TL;DR

This work presents a scalable paradigm for grounding visual concepts in real-world images by adaptively discovering image-specific concept axes with a pretrained vision-language model and a universal concept encoder that binds visual features to these axes. A compositional anchoring objective enforces axis-disentangled representations by ensuring edits affect only the targeted axis, while preserving instance-specific details. The approach, evaluated on ImageNet-S20, CelebA-HQ, and AFHQ, achieves superior editing quality and compositional generalization compared with prior language-informed and text-based editing methods, and demonstrates nuanced visual nuance transfer. By eliminating per-axis encoders and leveraging a single unified model, the framework enables flexible, scalable, and interpretable language-grounded visual concept learning suitable for diverse real-world domains. The codebase is publicly available, enabling reproducibility and broader adoption of adaptive VCL in real-world settings.

Abstract

Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. However, existing approaches to language-grounded visual concept learning are limited to a few predefined primitive axes, such as color and shape, and are typically explored in synthetic datasets. In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Leveraging a pretrained vision-language model and our universal prompting strategy, our framework identifies a diverse image-related axes without any prior knowledge. Our universal concept encoder adaptively binds visual features to the discovered axes without introducing additional model parameters for each concept. To ground visual concepts along the discovered axes, we optimize a compositional anchoring objective, which ensures that each axis can be independently manipulated without affecting others. We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ, and AFHQ, showcasing superior editing capabilities across diverse real-world concepts that are too varied to be manually predefined. Our method also exhibits strong compositional generalization, outperforming existing visual concept learning and text-based editing methods. The code is available at https://github.com/whieya/Language-grounded-VCL.

Figures (27)

• Figure 1: Overview of our method. Our framework first identifies image-related concept axes by leveraging VLM. We design an universal prompt that guides the VLM to find concept axes across different datasets. Given discovered axes for each image, our universal concept encoder binds visual features to those axes without introducing any specialized concept encoder for each axis. Finally, the encoded concept representations are regularized with a compositional anchoring loss to promote disentanglement between concept axes. Specifically, we randomly swap a concept representation with the one in an identical concept axis extracted from different images, and constrain composite images, rendered from randomly swapped representations, to be aligned with composite text descriptions.
• Figure 1: Comparisons on visual concept editing task. Our method consistently outperforms recent text-based editing methods meng2021sdeditmokady2023nulltextbrooks2023instructpix2pixhuberman2024ddpm_inversion and language-informed visual concept learning lee2023livcl.
• Figure 3: Compositional generalization to unseen concept combination. Given OOD combination of concepts such as frog with panda's fur pattern, only our method generates plausible results.
• Figure 3: Comparisons of visual concept editing. Our method outperforms recent text-based editing methods meng2021sdeditmokady2023nulltextbrooks2023instructpix2pixhuberman2024ddpm_inversion and language-informed visual concept learning lee2023livcl.
• Figure 5: Examples of visual nuance transfer. Even when transferring the same attributes, e.g., black and white fur or blonde hair, the outputs reflect subtle details of source images.