LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization

TL;DR

This work tackles small-data, large-scale stochastic optimization in operations management by introducing an LLM-inspired pretrain-then-finetune framework built on a problem-specific Transformer. It combines domain-informed synthetic pretraining with label-free finetuning on scarce real observations, enabled by a Stein-identity loss and LoRA parameter-efficient updates, to jointly leverage cross-task structure and domain knowledge. A comprehensive, nonasymptotic theory decomposes estimation error into domain gap, finetuning generalization, and approximation components, revealing an economy-of-scale effect: finetuning becomes more effective as the number of tasks $N$ grows, while informative pretraining can dominate when real data are extremely scarce. Simulation results on a large multi-product setting confirm the theoretical predictions, showing robust transfer across varied domain knowledge quality and target distributions, and demonstrating the practical viability of domain-guided pretraining for operational decision-making. Overall, the paper provides a principled framework and rigorous guarantees for Transformer-based pretrain-then-finetune in small-data, large-scale optimization and offers guidance for deploying domain-informed AI in real-world OM contexts.

Abstract

We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously (e.g., across a large product portfolio) while observing only a few, potentially noisy, data points per instance. Inspired by the success of large language models (LLMs), we propose a pretrain-then-finetune approach built on a designed Transformer model to address this challenge. The model is first pretrained on large-scale, domain-informed synthetic data that encode managerial knowledge and structural features of the decision environment, and is then fine-tuned on real observations. This new pipeline offers two complementary advantages: pretraining injects domain knowledge into the learning process and enables the training of high-capacity models using abundant synthetic data, while finetuning adapts the pretrained model to the operational environment and improves alignment with the true data-generating regime. While we have leveraged the Transformer's state-of-the-art representational capacity, particularly its attention mechanism, to efficiently extract cross-task structure, our approach is not an off-the-shelf application. Instead, it relies on problem-specific architectural design and a tailored training procedure to match the decision setting. Theoretically, we develop the first comprehensive error analysis regarding Transformer learning in relevant contexts, establishing nonasymptotic guarantees that validate the method's effectiveness. Critically, our analysis reveals how pretraining and fine-tuning jointly determine performance, with the dominant contribution governed by whichever is more favorable. In particular, finetuning exhibits an economies-of-scale effect, whereby transfer learning becomes increasingly effective as the number of instances grows.

Figures (6)

• Figure 1: The architecture of our adopted Transformer model.
• Figure 2: Illustration of our pretrain-then-finetune framework.
• Figure 3: Overview of our pretraining–finetuning error analysis framework.
• Figure 4: Excess risk of pretrained-only, pretrained-finetuned, and finetuned-from-scratch models versus the distributional distance between the pretraining and target distributions.
• Figure 5: Excess risk versus finetuning data size $N$ under four pretraining configurations.

Theorems & Definitions (11)

• Definition 1: Self-Attention Mechanism
• Proposition 1: Pretraining-Finetuning Interaction
• Theorem 1: Domain Gap
• Theorem 2: Finetuning Generalization Error
• Lemma 1: Kolmogorov-Arnold Representation Theorem kolmogorov1957representations
• Definition 2: Spectral Complexity
• Definition 3
• Theorem 3: Approximation Error
• Example 1: Large-Scale Newsvendor
• Example 2: Large-Scale Pricing