Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoLLEGe: Concept Embedding Generation for Large Language Models

Ryan Teehan, Brenden Lake, Mengye Ren

TL;DR

This paper introduces a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning, a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions.

Abstract

Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training. Code and data for our project can be found at https://college-concept-learning.github.io/

CoLLEGe: Concept Embedding Generation for Large Language Models

TL;DR

This paper introduces a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning, a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions.

Abstract

Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training. Code and data for our project can be found at https://college-concept-learning.github.io/
Paper Structure (47 sections, 3 equations, 2 figures, 13 tables)

This paper contains 47 sections, 3 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Few-Shot Word Learning:
  4. Meta-Learning:
  5. Compression:
  6. CoLLEGe: Concept Learning with Language Embedding Generation
  7. Concept Embedding Generation:
  8. Sampling Few-Shot Learning Episodes:
  9. Negative Examples:
  10. Knowledge Distillation:
  11. Final Loss:
  12. Datasets for Training CoLLEGe
  13. Experiments
  14. Implementation Details:
  15. Baselines:
  16. ...and 32 more sections

Figures (2)

  • Figure 1: Our model generates an embedding for an unseen token given one or a few example sentences. The ground truth word is pendant, and the model is able to generate an accurate definition using the embedding produced by CoLLEGe.
  • Figure 2: Our proposed CoLLEGe framework for concept embedding generation. Support sequences are embedded by a pretrained MLM (e.g. RoBERTa) with an additional Transformer encoder to produce pooled sequence embeddings for each support sequence. These are aggregated and projected into the input and output embedding space for the pretrained LLM (e.g. LLaMA). According to UrbanDictionary, beige flag, an Internet slang appeared in mid-2023, means "a benign but annoying trait or habit."