Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Language Generation Model: Loss Calibration and Formatted Decoding for Robust Structure Prediction and Knowledge Retrieval

Minho Lee, Junghyun Min, Yerang Kim, Woochul Lee, Yeonsoo Lee

TL;DR

<3-5 sentence high-level summary> The paper addresses the gap in structured prediction for Generative Pre-trained Language Models by diagnosing a misalignment between internal structural representations and the output space. It introduces the Structured Language Generation Model (SLGM), which couples reinforced input formatting, two targeted losses (Structure and Slot Loss) and format-aware decoding to constrain outputs to task-valid formats. Across 5 tasks and 13 datasets, SLGM improves structure prediction without extra parameters and remains competitive with much larger models when using sub-1B parameter sizes, also functioning as a zero-weight adapter in low-resource settings. The framework’s model- and data-agnostic design suggests broad applicability for structured prediction and knowledge retrieval in LLM-powered systems.

Abstract

Modern generative pre-trained language models excel at open-ended text generation, yet continue to underperform on structure-related tasks such as NER, relation extraction, and semantic role labeling, especially when compared to encoder-only models of similar sizes. While this gap has been attributed to limited structure knowledge, we hypothesize this is also due to the missing connection between the model's internal representations of linguistic structure and the output space used during supervised fine-tuning. We propose the Structured Language Generation Model (SLGM), a model- and task-agnostic framework that reformulates structured prediction as a classification problem through three components: (1) reinforced input formatting with structural cues, (2) loss design, and (3) format-aware decoding that constrains generation to task-valid outputs. Across 5 tasks and 13 datasets, SLGM substantially improves structure prediction without relying on dataset-specific engineering or additional model parameters, strengthening alignment between the model's internal structure representation and output. It outperforms baseline fine-tuning on models of the same size, achieves comparable performance to much larger models when used with <1B parameter models, and acts as a zero-weight adapter that reproduces the benefits of dataset-specific fine-tuning in low-resource settings.

Structured Language Generation Model: Loss Calibration and Formatted Decoding for Robust Structure Prediction and Knowledge Retrieval

TL;DR

<3-5 sentence high-level summary> The paper addresses the gap in structured prediction for Generative Pre-trained Language Models by diagnosing a misalignment between internal structural representations and the output space. It introduces the Structured Language Generation Model (SLGM), which couples reinforced input formatting, two targeted losses (Structure and Slot Loss) and format-aware decoding to constrain outputs to task-valid formats. Across 5 tasks and 13 datasets, SLGM improves structure prediction without extra parameters and remains competitive with much larger models when using sub-1B parameter sizes, also functioning as a zero-weight adapter in low-resource settings. The framework’s model- and data-agnostic design suggests broad applicability for structured prediction and knowledge retrieval in LLM-powered systems.

Abstract

Modern generative pre-trained language models excel at open-ended text generation, yet continue to underperform on structure-related tasks such as NER, relation extraction, and semantic role labeling, especially when compared to encoder-only models of similar sizes. While this gap has been attributed to limited structure knowledge, we hypothesize this is also due to the missing connection between the model's internal representations of linguistic structure and the output space used during supervised fine-tuning. We propose the Structured Language Generation Model (SLGM), a model- and task-agnostic framework that reformulates structured prediction as a classification problem through three components: (1) reinforced input formatting with structural cues, (2) loss design, and (3) format-aware decoding that constrains generation to task-valid outputs. Across 5 tasks and 13 datasets, SLGM substantially improves structure prediction without relying on dataset-specific engineering or additional model parameters, strengthening alignment between the model's internal structure representation and output. It outperforms baseline fine-tuning on models of the same size, achieves comparable performance to much larger models when used with <1B parameter models, and acts as a zero-weight adapter that reproduces the benefits of dataset-specific fine-tuning in low-resource settings.
Paper Structure (33 sections, 1 equation, 4 figures, 17 tables)

This paper contains 33 sections, 1 equation, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Structure Prediction and Knowledge Retrieval
  3. SLGM Framework
  4. Task-specific Output Format
  5. Format Loss
  6. Structure Loss.
  7. Slot Loss.
  8. Formatted Decoding during Inference
  9. Experiments
  10. Tasks and Datasets
  11. Structural Pre-training.
  12. Multi-task training.
  13. Baseline Models
  14. Evaluation Method
  15. Results and Discussion
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Sample NER example in DeepStructdeepstruct and SLGM. The yellow and navy highlight task specific and dataset information, respectively. SLGM breaks down the implicit information into explicit output format and tagset information.
  • Figure 2: Format error frequency and ratio for each task. Numbers indicate the total number of FEs across every dataset inside given task.
  • Figure 3: Average score and format error according to model size: small, base, and large. Solid line indicates Average F1 score (left y-axis). Dotted line indicates format errors (right y-axis).
  • Figure 4: Average score and format errors according to dataset fraction. Solid line represents model with structural pre-training, and dotted line represents model without structural pre-training.