Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FrameEOL: Semantic Frame Induction using Causal Language Models

Chihiro Yano, Kosuke Yamada, Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda

TL;DR

This work tackles semantic frame induction and introduces FrameEOL, a prompt-based method that prompts causal language models to output a single FrameNet frame name as the next token. Frame embeddings are refined using in-context learning and deep metric learning, followed by clustering to induce semantic frames. Empirical results on English FrameNet and Japanese FrameNet show that FrameEOL with DML outperforms MLM-based baselines, with particularly strong performance in low-resource Japanese where few-shot ICL approaches the efficacy of fine-tuned MLMs. The findings suggest that CLMs can effectively acquire frame knowledge with modest labeled data, offering a practical path for building frame resources in languages with limited resources.

Abstract

Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. In recent years, leveraging embeddings of frame-evoking words that are obtained using masked language models (MLMs) such as BERT has led to high-performance semantic frame induction. Although causal language models (CLMs) such as the GPT and Llama series succeed in a wide range of language comprehension tasks and can engage in dialogue as if they understood frames, they have not yet been applied to semantic frame induction. We propose a new method for semantic frame induction based on CLMs. Specifically, we introduce FrameEOL, a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label representing the given situation. To obtain embeddings more suitable for frame induction, we leverage in-context learning (ICL) and deep metric learning (DML). Frame induction is then performed by clustering the resulting embeddings. Experimental results on the English and Japanese FrameNet datasets demonstrate that the proposed methods outperform existing frame induction methods. In particular, for Japanese, which lacks extensive frame resources, the CLM-based method using only 5 ICL examples achieved comparable performance to the MLM-based method fine-tuned with DML.

FrameEOL: Semantic Frame Induction using Causal Language Models

TL;DR

This work tackles semantic frame induction and introduces FrameEOL, a prompt-based method that prompts causal language models to output a single FrameNet frame name as the next token. Frame embeddings are refined using in-context learning and deep metric learning, followed by clustering to induce semantic frames. Empirical results on English FrameNet and Japanese FrameNet show that FrameEOL with DML outperforms MLM-based baselines, with particularly strong performance in low-resource Japanese where few-shot ICL approaches the efficacy of fine-tuned MLMs. The findings suggest that CLMs can effectively acquire frame knowledge with modest labeled data, offering a practical path for building frame resources in languages with limited resources.

Abstract

Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. In recent years, leveraging embeddings of frame-evoking words that are obtained using masked language models (MLMs) such as BERT has led to high-performance semantic frame induction. Although causal language models (CLMs) such as the GPT and Llama series succeed in a wide range of language comprehension tasks and can engage in dialogue as if they understood frames, they have not yet been applied to semantic frame induction. We propose a new method for semantic frame induction based on CLMs. Specifically, we introduce FrameEOL, a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label representing the given situation. To obtain embeddings more suitable for frame induction, we leverage in-context learning (ICL) and deep metric learning (DML). Frame induction is then performed by clustering the resulting embeddings. Experimental results on the English and Japanese FrameNet datasets demonstrate that the proposed methods outperform existing frame induction methods. In particular, for Japanese, which lacks extensive frame resources, the CLM-based method using only 5 ICL examples achieved comparable performance to the MLM-based method fine-tuned with DML.

Paper Structure

This paper contains 29 sections, 1 equation, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Frame Induction
  4. Prompt-Based Text Embeddings
  5. Proposed Method
  6. FrameEOL
  7. Frame Embedding Optimization via ICL
  8. Frame Embedding Optimization via DML
  9. Experiments on FrameNet
  10. Evaluation Setup
  11. Dataset
  12. Clustering Methods
  13. Evaluation Methods
  14. Training Settings
  15. Comparison Methods
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Examples of ChatGPT output showing the possibility of recognizing differences in semantic frames evoked by the same predicate lost.
  • Figure 2: Overview of the frame induction process using FrameEOL and its evaluation.
  • Figure 3: Example of input in 3-shot ICL setting.
  • Figure 4: Overview of DML: embeddings of the same frame are trained to be closer, while embeddings of different frames are trained to be farther apart.