Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning

Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yunlong Yu, Xi Li

TL;DR

The paper tackles the challenge of few-shot learning by enriching visual-class prototypes with both abstract semantics and concrete class entities generated by large language models. It proposes a two-stage training framework: multi-modality pre-training to fuse visual and textual cues, and a fine-tuning stage that uses LLM-derived entities within a Progressive Visual-Semantic Aggregation (PVSA) pipeline, comprising SVPE for semantic-guided pattern extraction and PC for prototype calibration. The method uses a dual-level visual-language objective during pre-training and a carefully designed entity-filtering strategy to curb LLM hallucinations, achieving state-of-the-art results on multiple FSL and cross-domain benchmarks, especially in 1-shot tasks. The approach demonstrates strong improvements in discriminability and generalization, with practical impact for low-data visual recognition scenarios where rich semantic priors can be leveraged to compensate for scarce labeled examples.

Abstract

Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.

Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning

TL;DR

The paper tackles the challenge of few-shot learning by enriching visual-class prototypes with both abstract semantics and concrete class entities generated by large language models. It proposes a two-stage training framework: multi-modality pre-training to fuse visual and textual cues, and a fine-tuning stage that uses LLM-derived entities within a Progressive Visual-Semantic Aggregation (PVSA) pipeline, comprising SVPE for semantic-guided pattern extraction and PC for prototype calibration. The method uses a dual-level visual-language objective during pre-training and a carefully designed entity-filtering strategy to curb LLM hallucinations, achieving state-of-the-art results on multiple FSL and cross-domain benchmarks, especially in 1-shot tasks. The approach demonstrates strong improvements in discriminability and generalization, with practical impact for low-data visual recognition scenarios where rich semantic priors can be leveraged to compensate for scarce labeled examples.

Abstract

Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.
Paper Structure (17 sections, 11 equations, 8 figures, 4 tables)

This paper contains 17 sections, 11 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Uni-Modal Few-Shot Learning
  4. Multi-Modal Few-Shot Learning
  5. Methodology
  6. Multi-Modality Pre-training
  7. Entities-Assisted Episode-based Fine-Tuning
  8. Progressive Visual-Semantic Aggregation
  9. Semantic-guided Visual Pattern Extraction (SVPE).
  10. Prototype Calibration.
  11. Objective Function.
  12. Experiments
  13. Experimental Details
  14. Performance Comparison
  15. Ablation Study
  16. ...and 2 more sections

Figures (8)

  • Figure 1: The key idea is to learn a classifier from data using a visual sample per category, extended with additional visual patterns derived from semantic entities generated by LLMs.
  • Figure 2: Multi-Modality Pre-training Stage. Our pre-training paradigm trains a visual encoder that captures more semantic-rich information under the guidance of the textual captions derived from an off-the-shelf model, which harnesses the power of natural language processing to enrich the visual representation, fostering a deeper understanding of the images and their underlying semantics.
  • Figure 3: The framework of our progressive visual-semantic aggregation (PVSA) leverages both the class name ("Newfoundland") and class-related entities ("Thick Coat", "Black Fur") to enrich the visual prototypes. PVSA consists of a semantic-guided visual pattern extraction (SVPE) module that extracts visual patterns that are related to the class semantics and a prototype calibration (PC) module that enriches the visual prototype by incorporating the semantic-aware visual features.
  • Figure 4: The entities produced with LLMs prompt and filtered with the proposed selection strategy.
  • Figure 5: Effects (%) of entity number and SVPE stages.
  • ...and 3 more figures