Scalable Representation Learning for Multimodal Tabular Transactions

TL;DR

This paper introduces a multi-tier partitioning mechanism that utilizes power-law dynamics to handle large vocabularies, an adaptive quantization mechanism to impose priors on numerical continuity, and a distinct treatment of core-columns and meta-information columns to facilitate instruction tuning on LLMs.

Abstract

Large language models (LLMs) are primarily designed to understand unstructured text. When directly applied to structured formats such as tabular data, they may struggle to discern inherent relationships and overlook critical patterns. While tabular representation learning methods can address some of these limitations, existing efforts still face challenges with sparse high-cardinality fields, precise numerical reasoning, and column-heavy tables. Furthermore, leveraging these learned representations for downstream tasks through a language based interface is not apparent. In this paper, we present an innovative and scalable solution to these challenges. Concretely, our approach introduces a multi-tier partitioning mechanism that utilizes power-law dynamics to handle large vocabularies, an adaptive quantization mechanism to impose priors on numerical continuity, and a distinct treatment of core-columns and meta-information columns. To facilitate instruction tuning on LLMs, we propose a parameter efficient decoder that interleaves transaction and text modalities using a series of adapter layers, thereby exploiting rich cross-task knowledge. We validate the efficacy of our solution on a large-scale dataset of synthetic payments transactions.

Figures (8)

• Figure 1: Scalable multimodal foundation model for transaction data. (1) Sparse yet large vocabulary categorical features are learned using multi-tier partitions. (2) Meta-column features are pre-learned offline, with only selected segments integrated. (3) Numerical values are quantized into a coarse vocabulary to emphasize ranges. (4) Compact transaction embeddings are interleaved with verbose instruction text. (5) A limited set of alignment parameters are fine-tuned for task-specific adaptation.
• Figure 2: Preferential partitioning of the embedding space. Left: Embedding parameter size can increase dramatically with a large vocabulary. Right: A non-uniform partitioning mechanism where a preferred bin with fewer vocabulary items is allocated a larger portion of the embedding space.
• Figure 3: Instruction tuning workflow. Transaction records, textual instructions, and task information are combined and fed into an LLM. All layers are frozen (shown in blue) except for the adapter layers (shown in green). These adapter layers learn to align the transaction and text embeddings with the specified task, enabling the LLM to generate response instructions.
• Figure 4: Sample synthetic transaction record.
• Figure 5: Mixture models in plate notation. These models are used for sampling the number of accounts in a company, the number of accounts a given account can transact with, the number of transactions between a (source, target) pair, and the amount and date values.