Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics

Gyanendra Shrestha, Anna Pyayt, Michael Gubanov

TL;DR

This paper proposes CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance and introduces a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics.

Abstract

Large pre-trained models (LMs) and Large Language Models (LLMs) are typically effective at capturing language semantics and contextual relationships. However, these models encounter challenges in maintaining optimal performance on tasks involving numbers. Blindly treating numerical or structured data as terms is inadequate -- their semantics must be well understood and encoded by the models. In this paper, we propose CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance. We introduce a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics. We conduct extensive experimental evaluation on large-scale datasets across diverse domains (web, medical, finance, and government) that justifies CONE's strong numerical reasoning capabilities, achieving an F1 score of 87.28% on DROP, a remarkable improvement of up to 9.37% in F1 over state-of-the-art (SOTA) baselines, and outperforming major SOTA models with a significant Recall@10 gain of up to 25%.

CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics

TL;DR

This paper proposes CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance and introduces a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics.

Abstract

Large pre-trained models (LMs) and Large Language Models (LLMs) are typically effective at capturing language semantics and contextual relationships. However, these models encounter challenges in maintaining optimal performance on tasks involving numbers. Blindly treating numerical or structured data as terms is inadequate -- their semantics must be well understood and encoded by the models. In this paper, we propose CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance. We introduce a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics. We conduct extensive experimental evaluation on large-scale datasets across diverse domains (web, medical, finance, and government) that justifies CONE's strong numerical reasoning capabilities, achieving an F1 score of 87.28% on DROP, a remarkable improvement of up to 9.37% in F1 over state-of-the-art (SOTA) baselines, and outperforming major SOTA models with a significant Recall@10 gain of up to 25%.
Paper Structure (32 sections, 16 equations, 12 figures, 7 tables, 2 algorithms)

This paper contains 32 sections, 16 equations, 12 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. CONE Model
  4. Encoder
  5. Embeddings for Numerical Values
  6. Composite Embedding Structure
  7. Datasets
  8. Analysis of Distance
  9. Analytic distance metrics
  10. Numbers (absolute difference)
  11. Ranges
  12. Gaussians (2-Wasserstein distance)
  13. CONE Embedding Components
  14. Numbers
  15. Ranges
  16. ...and 17 more sections

Figures (12)

  • Figure 1: Distance Between Age and Other Attributes, Calculated Using BioBERT and CoNE.
  • Figure 2: A Table with Numerical Data Including Ranges and Gaussians Accompanied by Units.
  • Figure 3: CONE Architecture including the Encoder and Numerical Value Embeddings. The Encoder Embeddings for Numbers ($M^E$, shown in blue) are Fused with Numerical Value Embeddings ($M^N$, pink) and Refined Through a Transformer to Produce the Contextualized Representations ($M^O$, green).
  • Figure 4: Composite Embedding Structure for (a) Complex Numerical Data, (b) Numerical Ranges, and (c) Gaussians. T: Text, N: Numerical Value, SD: Standard Deviation.
  • Figure 5: Euclidean Distance from Embedding of 0 to the Embeddings of Numbers from 0 to 100
  • ...and 7 more figures