Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feasibility with Language Models for Open-World Compositional Zero-Shot Learning

Jae Myung Kim, Stephan Alaniz, Cordelia Schmid, Zeynep Akata

TL;DR

The paper tackles open-world compositional zero-shot learning by predicting the feasibility of state-object pairs using large language models. It introduces FLM, which uses in-context guided prompts and the logit of 'Yes' from an LLM to score feasibility and prune the label space before CZSL classification. Across MIT-States, UT-Zappos, and C-GQA, FLM consistently improves performance of CLIP-based VLMs and outperforms previous feasibility baselines, with Vicuna and ChatGPT performing especially well. The work demonstrates that few-shot, context-aware LLM prompts can capture nuanced semantic feasibility and boost open-world recognition, while also highlighting considerations around prompt design and potential training-data biases.

Abstract

Humans can easily tell if an attribute (also called state) is realistic, i.e., feasible, for an object, e.g. fire can be hot, but it cannot be wet. In Open-World Compositional Zero-Shot Learning, when all possible state-object combinations are considered as unseen classes, zero-shot predictors tend to perform poorly. Our work focuses on using external auxiliary knowledge to determine the feasibility of state-object combinations. Our Feasibility with Language Model (FLM) is a simple and effective approach that leverages Large Language Models (LLMs) to better comprehend the semantic relationships between states and objects. FLM involves querying an LLM about the feasibility of a given pair and retrieving the output logit for the positive answer. To mitigate potential misguidance of the LLM given that many of the state-object compositions are rare or completely infeasible, we observe that the in-context learning ability of LLMs is essential. We present an extensive study identifying Vicuna and ChatGPT as best performing, and we demonstrate that our FLM consistently improves OW-CZSL performance across all three benchmarks.

Feasibility with Language Models for Open-World Compositional Zero-Shot Learning

TL;DR

The paper tackles open-world compositional zero-shot learning by predicting the feasibility of state-object pairs using large language models. It introduces FLM, which uses in-context guided prompts and the logit of 'Yes' from an LLM to score feasibility and prune the label space before CZSL classification. Across MIT-States, UT-Zappos, and C-GQA, FLM consistently improves performance of CLIP-based VLMs and outperforms previous feasibility baselines, with Vicuna and ChatGPT performing especially well. The work demonstrates that few-shot, context-aware LLM prompts can capture nuanced semantic feasibility and boost open-world recognition, while also highlighting considerations around prompt design and potential training-data biases.

Abstract

Humans can easily tell if an attribute (also called state) is realistic, i.e., feasible, for an object, e.g. fire can be hot, but it cannot be wet. In Open-World Compositional Zero-Shot Learning, when all possible state-object combinations are considered as unseen classes, zero-shot predictors tend to perform poorly. Our work focuses on using external auxiliary knowledge to determine the feasibility of state-object combinations. Our Feasibility with Language Model (FLM) is a simple and effective approach that leverages Large Language Models (LLMs) to better comprehend the semantic relationships between states and objects. FLM involves querying an LLM about the feasibility of a given pair and retrieving the output logit for the positive answer. To mitigate potential misguidance of the LLM given that many of the state-object compositions are rare or completely infeasible, we observe that the in-context learning ability of LLMs is essential. We present an extensive study identifying Vicuna and ChatGPT as best performing, and we demonstrate that our FLM consistently improves OW-CZSL performance across all three benchmarks.
Paper Structure (22 sections, 6 equations, 6 figures, 4 tables)

This paper contains 22 sections, 6 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. In-context Feasibility Prediction Framework
  4. Open-World Compositional Zero-Shot Learning (OW-CZSL)
  5. Feasibility with Language Model (FLM)
  6. Experiments
  7. Experimental setup
  8. LLM-generated feasibility in OW-CZSL
  9. Ablation Study
  10. Conclusion
  11. Broader Impact and Limitations
  12. Prompt search in FLM
  13. Implementation Details
  14. Datasets
  15. Evaluation Metrics
  16. ...and 7 more sections

Figures (6)

  • Figure 1: The pipeline of our Feasibility with Language Model (FLM) method. We constrict a prompt containing a list of related seen classes from the training set and a query to classify an unssen state-object pair as feasible. By comparing the LLM logit for the token "Yes" with a threshold $\tau$ we determine whether a pair is feasible, in which case it is used for OW-CZSL classification.
  • Figure 2: Comparison of FLM using Vicuna, LLaMA-2, and proprietary models (GPT-4, PaLM-2, Claude-2, ChatGPT) as LLMs on MIT-States and UT-Zappos. Proprietary models can only provide a binary "Yes" or "No" response, whereas for Vicuna and LLaMA-2 we evaluate both the binary and logit outputs as feasibility scores.
  • Figure 3: Prompt variation results on MIT-States dataset.
  • Figure 4: Feasibility scores for question-answer "yes" format in the in-context prompt.
  • Figure 5: Distributions of feasibility scores of all state-object pairs. For best separation, feasible classes should be close to 1 and all remaining confusing classes close to 0.
  • ...and 1 more figures