Small Models are LLM Knowledge Triggers on Medical Tabular Prediction

TL;DR

The paper targets the challenge of applying large language models to numeric tabular data, proposing SERSAL, an unsupervised self-prompting loop that pairs LLM-derived soft pseudo-labels with a small tabular model (FT-Transformer) in a teacher-student dynamic. Through soft labeling, semi-supervised learning with noisy labels (DivideMix), and reverse LLM tuning, SERSAL enables iterative refinement of both the small model and the LLM without gold labels. Empirical results on ten medical tabular datasets show SERSAL consistently outperforms traditional prompting baselines, with greater gains as LLM capability increases (e.g., GPT-4), and extended improvements via multi-loop iterations. The approach also demonstrates cross-domain applicability and interpretability via Shapley-value analysis, suggesting a general, zero-label pathway to extend LLM knowledge to structured data tasks.

Abstract

Recent development in large language models (LLMs) has demonstrated impressive domain proficiency on unstructured textual or multi-modal tasks. However, despite with intrinsic world knowledge, their application on structured tabular data prediction still lags behind, primarily due to the numerical insensitivity and modality discrepancy that brings a gap between LLM reasoning and statistical tabular learning. Unlike textual or vision data (e.g., electronic clinical notes or medical imaging data), tabular data is often presented in heterogeneous numerical values (e.g., CBC reports). This ubiquitous data format requires intensive expert annotation, and its numerical nature limits LLMs' capability to effectively transfer untapped domain expertise. In this paper, we propose SERSAL, a general self-prompting method by synergy learning with small models to enhance LLM tabular prediction in an unsupervised manner. Specifically, SERSAL utilizes the LLM's prior outcomes as original soft noisy annotations, which are dynamically leveraged to teach a better small student model. Reversely, the outcomes from the trained small model are used to teach the LLM to further refine its real capability. This process can be repeatedly applied to gradually distill refined knowledge for continuous progress. Comprehensive experiments on widely used medical domain tabular datasets show that, without access to gold labels, applying SERSAL to OpenAI GPT reasoning process attains substantial improvement compared to linguistic prompting methods, which serves as an orthogonal direction for tabular LLM, and increasing prompting bonus is observed as more powerful LLMs appear.

Figures (4)

• Figure 1: (a) Comparison of prompting effectiveness on unstructured textual data mullenbach2018explainable and structured tabular data detrano1989international from medical domain, it is clearly seen, even with surprising medical expertise nori2023can, GPT-4 still struggles to catch up fully supervised small models (ClinicalBERT huang2019clinicalbert for textual tasks and FT-Transformer gorishniy2021revisiting for tabular ones) on tabular data, implying essential task discrepancy that makes it incompatible to rely on typical prompting techniques to unlock the potential of LLMs for tabular prediction. (b) Unsupervised SERSAL triggers LLM's knowledge using a small model.
• Figure 2: Performances in different LLM's confidence ranges on ECD and LI datasets. Extreme-confidence samples are relatively more reliable.
• Figure 3: Interpretability visualization from feature importance perspective: the variation of the Shapley Values (treat SERSAL outputs as the targets) and performances on Indian Liver Patient Records (LI dataset) after one and two SERSAL loops using GPT-3.5.
• Figure 4: Performances in different LLM confidence ranges on other eight datasets. The overall trend of high-confidence samples being relatively more reliable still holds.