Table Transformers for Imputing Textual Attributes

TL;DR

A novel end-to-end approach called TTITA based on the transformer to impute unstructured textual columns using other columns in the table based on the transformer to impute unstructured textual columns using other columns in the table is proposed.

Abstract

Missing data in tabular dataset is a common issue as the performance of downstream tasks usually depends on the completeness of the training dataset. Previous missing data imputation methods focus on numeric and categorical columns, but we propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) based on the transformer to impute unstructured textual columns using other columns in the table. We conduct extensive experiments on three datasets, and our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. The performance improvement is more significant when the target sequence has a longer length. Additionally, we incorporate multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation. We also qualitatively compare with ChatGPT for realistic applications.

Figures (4)

• Figure 1: The architecture of the proposed TTITA Model. Input columns are encoded as the following: numeric columns connect to an FC layer of dimension $d_n$, categorical columns connect to an embedding layer and then an FC layer of size $d_c$, and textual columns are featurized by a hashing vectorizer of dimension $d_t$. The latent representations are concatenated and fed into TTITA's decoder for cross-attention, which generates the imputed sequence in an auto-regressive manner.
• Figure 2: Structure of multi-task learning. We adapt TTITA to a multi-task learning framework that trains to simultaneously impute multiple columns of numeric, textual, or categorical types.
• Figure 3: Model Parameter Count vs. Inference Speed. While being slower than LSTM and GRU, TTITA is comparable to Mode imputation and faster than KNN. Circle size is proportional to inference speed and RNN denotes LSTM and GRU.
• Figure 4: Hyperparameter Analysis on Input Column Embedding Size. We investigate different embedding sizes for numeric ($d_n$), categorical ($d_c$), and textual ($d_t$) input columns. $d_n=100$, $d_c=10$, and $d_t=128$ resulted in better performance.