Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LBC: Language-Based-Classifier for Out-Of-Variable Generalization

Kangjun Noh, Baekryun Seong, Hoyoon Byun, Youngjun Choi, Sungjin Song, Kyungwoo Song

TL;DR

This work addresses Out-of-Variable (OOV) generalization in tabular data by introducing Language-Based-Classifier (LBC), which leverages pre-trained LLMs through prompt-based inference. LBC combines three strategies—Categorical Change (converting numerical features to $N$-category bins), Advanced Order and Indicator (careful prompt sequencing and indicators to separate IVs from OOVs), and Verbalizer (mapping logits to class scores via central words and synonyms with a tailored loss)—and utilizes LoRA for efficient fine-tuning. Empirically, LBC outperforms traditional ML baselines in binary and multiclass OOV tasks across multiple datasets, and theoretically, LoRA-based fine-tuning can approximate arbitrary classifiers, supporting robust generalization. The findings highlight the practical potential of LLMs for OOV tabular tasks, with open-source code and evidence that advanced prompting substantially boosts performance, including when using black-box LLMs via in-context learning.

Abstract

Large Language Models (LLMs) have great success in natural language processing tasks such as response generation. However, their use in tabular data has been limited due to their inferior performance compared to traditional machine learning models (TMLs) such as XGBoost. We find that the pre-trained knowledge of LLMs enables them to interpret new variables that appear in a test without additional training, a capability central to the concept of Out-of-Variable (OOV). From the findings, we propose a Language-Based-Classifier (LBC), a classifier that maximizes the benefits of LLMs to outperform TMLs on OOV tasks. LBC employs three key methodological strategies: 1) Categorical changes to adjust data to better fit the model's understanding, 2) Advanced order and indicator to enhance data representation to the model, and 3) Using verbalizer to map logit scores to classes during inference to generate model predictions. These strategies, combined with the pre-trained knowledge of LBC, emphasize the model's ability to effectively handle OOV tasks. We empirically and theoretically validate the superiority of LBC. LBC is the first study to apply an LLM-based model to OOV tasks. The source code is at https://github.com/sksmssh/LBCforOOVGen

LBC: Language-Based-Classifier for Out-Of-Variable Generalization

TL;DR

This work addresses Out-of-Variable (OOV) generalization in tabular data by introducing Language-Based-Classifier (LBC), which leverages pre-trained LLMs through prompt-based inference. LBC combines three strategies—Categorical Change (converting numerical features to -category bins), Advanced Order and Indicator (careful prompt sequencing and indicators to separate IVs from OOVs), and Verbalizer (mapping logits to class scores via central words and synonyms with a tailored loss)—and utilizes LoRA for efficient fine-tuning. Empirically, LBC outperforms traditional ML baselines in binary and multiclass OOV tasks across multiple datasets, and theoretically, LoRA-based fine-tuning can approximate arbitrary classifiers, supporting robust generalization. The findings highlight the practical potential of LLMs for OOV tabular tasks, with open-source code and evidence that advanced prompting substantially boosts performance, including when using black-box LLMs via in-context learning.

Abstract

Large Language Models (LLMs) have great success in natural language processing tasks such as response generation. However, their use in tabular data has been limited due to their inferior performance compared to traditional machine learning models (TMLs) such as XGBoost. We find that the pre-trained knowledge of LLMs enables them to interpret new variables that appear in a test without additional training, a capability central to the concept of Out-of-Variable (OOV). From the findings, we propose a Language-Based-Classifier (LBC), a classifier that maximizes the benefits of LLMs to outperform TMLs on OOV tasks. LBC employs three key methodological strategies: 1) Categorical changes to adjust data to better fit the model's understanding, 2) Advanced order and indicator to enhance data representation to the model, and 3) Using verbalizer to map logit scores to classes during inference to generate model predictions. These strategies, combined with the pre-trained knowledge of LBC, emphasize the model's ability to effectively handle OOV tasks. We empirically and theoretically validate the superiority of LBC. LBC is the first study to apply an LLM-based model to OOV tasks. The source code is at https://github.com/sksmssh/LBCforOOVGen
Paper Structure (32 sections, 2 theorems, 11 equations, 9 figures, 5 tables)

This paper contains 32 sections, 2 theorems, 11 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Tabular Data Analysis with LLMs
  4. Out-of-Variable
  5. Verbalizer
  6. Low-Rank Adaption
  7. Preliminary
  8. Basic Dataset Conversion
  9. The Order of the Variables
  10. Fine-tuning LLM
  11. LLM-based Tabular Prediction
  12. Methodology
  13. Categorical Change
  14. The Advanced Order and Indicator
  15. Generalization Ability of LBC: LoRA
  16. ...and 17 more sections

Key Result

Theorem 1

Let $f(\boldsymbol{x})$ represents the ReLU neural network to which LoRA is applied, with no activation function in the last layer, and $\Bar{f}(\boldsymbol{x})$ represents the target single-layer linear network. Let $g(x)$ is the logistic function $(1+e^{-x})^{-1}$. $\sigma(\boldsymbol{W})_i$ is th

Figures (9)

  • Figure 1: (a) Illustration of OOV task. The variables that were not present in the training data appear in the test data. (b) Key components of LBC to increase performance in OOV tasks. Categorical change refines data to make it easier for LBC to interpret. The advanced order and indicator method enhances the prompts that feed into LBC. The verbalizer aggregates the probabilities for a particular class scattered throughout the logit score and maps them to a specific class.
  • Figure 2: The overall process of an LBC performing an OOV task. LBC transforms tabular data into advanced prompt (AP) utilizing strategies that are 1) Categorical change and 2) Advanced order and indicator. These APs are then input into an LLM that has been fine-tuned with a LoRA adapter, to derive a logit score for the answer token. This logit score is assessed against the label to calculate loss, and during inference, the model prediction is generated by mapping the logit score to a class via a 3) Verbalizer.
  • Figure 3: LIFT vs LBC in 50% randomly selected OOV situation. Both LLMs have a performance improvement when LBC's methodologies are applied rather than LIFT.
  • Figure 4: Graph of accuracy changing over OOV ratio (%): We observed the accuracy change of TMLs and LBCs by increasing the OOV ratio from 0, 30, 50, and 70 (%) for four datasets. Comparing the accuracy reduction of TMLs and LBCs, the reduction of LBCs is smaller compared to TMLs. It demonstrates that LBCs interpret OOVs, unlike TMLs.
  • Figure 5: Observing how LBC applies its pre-trained knowledge to prompts about OOVs, thereby revealing biases in its pre-trained knowledge. Intuitively, LBC has a bias toward making its predictions closer to the correct answer. However, it is not responsive to special variable names that do not have a word meaning.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Lemma 1