Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Active In-Context Learning for Tabular Foundation Models

Wilailuck Treerath, Fabrizio Pittorino

Abstract

Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Active In-Context Learning for Tabular Foundation Models

Abstract

Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Paper Structure

This paper contains 28 sections, 2 equations, 5 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Tabular Deep Learning and Foundation Models
  4. Active Learning Challenges
  5. Foundation Models and Active Learning
  6. Tabular Active In-Context Learning
  7. Problem Formulation
  8. Data Preprocessing
  9. Acquisition Strategies
  10. Margin Sampling (TabPFN-Margin)
  11. Hybrid Sampling (TabPFN-Hybrid)
  12. Proxy-Hybrid Sampling (TabPFN-Proxy-Hybrid)
  13. Coreset Sampling (Diversity)
  14. Experimental Setup
  15. Active Learning Protocol
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Comprehensive AULC Performance: Area Under Learning Curve scores across 20 datasets, grouped by acquisition strategy.
  • Figure 2: Learning Curve for Ionosphere: Tab-AICL strategies achieve rapid convergence, with TabPFN-Proxy-Hybrid offering an effective balance of speed and stability. The shaded regions represent 95% confidence intervals across 10 random seeds.
  • Figure 3: Learning Curve for Adult: The TabPFN-Proxy-Hybrid strategy (pink) demonstrates stable performance and consistent gains. Note the narrower confidence intervals compared to other strategies.
  • Figure 4: Learning Curve for Vehicle: The TabPFN-Hybrid strategy (purple) achieves higher performance than pure uncertainty sampling and baselines.
  • Figure 5: Learning Curve for Page Blocks: The TabPFN-Margin strategy (blue) achieves rapid convergence, suggesting that for certain data manifolds, pure uncertainty sampling remains an effective approach.