Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Object Language Modeling (SoLM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising

Amir Tavanaei, Kee Kiat Koo, Hayreddin Ceker, Shaobai Jiang, Qi Li, Julien Han, Karim Bouyarmane

TL;DR

The paper tackles the challenge of generating structured objects that strictly conform to complex schemas with interdependent facets. It introduces Structured Object Language Model (SoLM), a decoder-based system trained in two stages: self-supervised denoising using targeted noising functions on a large e-commerce product catalog, followed by supervised fine-tuning with human demonstrations. SoLM achieves performance on par with or better than prompt-engineered SOTA models while offering an order-of-magnitude reduction in compute cost, enabling single-pass generation without manual prompts or schema pre-specifications. Across offline, real-case, and online evaluations, SoLM improves precision, reduces hallucination, and yields measurable business impact such as revenue uplifts in online A/B tests. This work demonstrates a production-ready pathway for native structured-object generation with broad applicability to APIs and data stores in commerce and beyond.

Abstract

In this paper, we study the problem of generating structured objects that conform to a complex schema, with intricate dependencies between the different components (facets) of the object. The facets of the object (attributes, fields, columns, properties) can be a mix of short, structured, type-constrained facts, or long natural-language descriptions. The object has to be self-consistent between the different facets in the redundant information it carries (relative consistency), while being grounded with respect to world knowledge (absolute consistency). We frame the problem as a Language Modeling problem (Structured Object Language Modeling) and train an LLM to perform the task natively, without requiring instructions or prompt-engineering. We propose a self-supervised denoising method to train the model from an existing dataset of such objects. The input query can be the existing object itself, in which case the model acts as a regenerator, completing, correcting, normalizing the input, or any unstructured blurb to be structured. We show that the self-supervised denoising training provides a strong baseline, and that additional supervised fine-tuning with small amount of human demonstrations leads to further improvement. Experimental results show that the proposed method matches or outperforms prompt-engineered general-purpose state-of-the-art LLMs (Claude 3, Mixtral-8x7B), while being order-of-magnitude more cost-efficient.

Structured Object Language Modeling (SoLM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising

TL;DR

The paper tackles the challenge of generating structured objects that strictly conform to complex schemas with interdependent facets. It introduces Structured Object Language Model (SoLM), a decoder-based system trained in two stages: self-supervised denoising using targeted noising functions on a large e-commerce product catalog, followed by supervised fine-tuning with human demonstrations. SoLM achieves performance on par with or better than prompt-engineered SOTA models while offering an order-of-magnitude reduction in compute cost, enabling single-pass generation without manual prompts or schema pre-specifications. Across offline, real-case, and online evaluations, SoLM improves precision, reduces hallucination, and yields measurable business impact such as revenue uplifts in online A/B tests. This work demonstrates a production-ready pathway for native structured-object generation with broad applicability to APIs and data stores in commerce and beyond.

Abstract

In this paper, we study the problem of generating structured objects that conform to a complex schema, with intricate dependencies between the different components (facets) of the object. The facets of the object (attributes, fields, columns, properties) can be a mix of short, structured, type-constrained facts, or long natural-language descriptions. The object has to be self-consistent between the different facets in the redundant information it carries (relative consistency), while being grounded with respect to world knowledge (absolute consistency). We frame the problem as a Language Modeling problem (Structured Object Language Modeling) and train an LLM to perform the task natively, without requiring instructions or prompt-engineering. We propose a self-supervised denoising method to train the model from an existing dataset of such objects. The input query can be the existing object itself, in which case the model acts as a regenerator, completing, correcting, normalizing the input, or any unstructured blurb to be structured. We show that the self-supervised denoising training provides a strong baseline, and that additional supervised fine-tuning with small amount of human demonstrations leads to further improvement. Experimental results show that the proposed method matches or outperforms prompt-engineered general-purpose state-of-the-art LLMs (Claude 3, Mixtral-8x7B), while being order-of-magnitude more cost-efficient.

Paper Structure

This paper contains 19 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Self-Supervised Training
  3. Noising Functions
  4. Training Data Preparation
  5. Denoising Training
  6. Supervised Fine Tuning
  7. Data Corpus
  8. Self-Supervised Denoising Dataset
  9. SFT Dataset
  10. Model Training and Evaluation Metrics
  11. Training
  12. Evaluation Metrics
  13. Experiments and Results
  14. Offline Evaluations
  15. Real Test Cases
  16. ...and 4 more sections

Figures (3)

  • Figure 1: The noising functions applied to structured objects of products in e-commerce.
  • Figure 2: A structured object representing a product listing that includes multiple correlated components.
  • Figure 3: Zero-shot Mixtral with short prompt versus the self-supervised SoLM-7B for free-form text generation representing Title, Feature Bullets, and Product Description components of product listings in e-commerce.