Breaking the Gradient Barrier: Unveiling Large Language Models for Strategic Classification

TL;DR

GLIM presents a gradient-free approach to strategic classification by embedding the bi-level Stackelberg optimization into pre-trained LLMs through in-context learning. The authors show, both theoretically and empirically, that ICL can implicitly simulate both the inner-stage strategic manipulation and the outer-stage decision-rule optimization without fine-tuning, effectively matching gradient-based updates in a forward pass. Across six datasets spanning finance and internet domains, GLIM demonstrates robustness, scalability, and competitive accuracy under strategic manipulation, while highlighting practical considerations like prompt costs and API usage. This work bridges strategic ML and LLMs, offering a retraining-free, scalable pathway for large-scale SC with interpretable attention-based insights.

Abstract

Strategic classification~(SC) explores how individuals or entities modify their features strategically to achieve favorable classification outcomes. However, existing SC methods, which are largely based on linear models or shallow neural networks, face significant limitations in terms of scalability and capacity when applied to real-world datasets with significantly increasing scale, especially in financial services and the internet sector. In this paper, we investigate how to leverage large language models to design a more scalable and efficient SC framework, especially in the case of growing individuals engaged with decision-making processes. Specifically, we introduce GLIM, a gradient-free SC method grounded in in-context learning. During the feed-forward process of self-attention, GLIM implicitly simulates the typical bi-level optimization process of SC, including both the feature manipulation and decision rule optimization. Without fine-tuning the LLMs, our proposed GLIM enjoys the advantage of cost-effective adaptation in dynamic strategic environments. Theoretically, we prove GLIM can support pre-trained LLMs to adapt to a broad range of strategic manipulations. We validate our approach through experiments with a collection of pre-trained LLMs on real-world and synthetic datasets in financial and internet domains, demonstrating that our GLIM exhibits both robustness and efficiency, and offering an effective solution for large-scale SC tasks.

Figures (7)

• Figure 1: The figure illustrates a strategic classification scenario. Comparison between traditional gradient-based approaches and our gradient-free method using LLMs with ICL for efficient adaptation to Large-scale and evolving data without fine-tuning.
• Figure 2: Bi-level optimization in strategic classification is simulated within LLMs, where both inner and outer stage optimizations are realized via ICL.
• Figure 3: Comparison of ICL-guided strategic manipulation. (a) and (b) compare ICL and gradient-descent methods across data scales; (c) and (d) evaluate implicit gradient alignment via distribution metrics.
• Figure 4: Comparison of ICL-guided decision rule optimization with Linear and non-linear self-attention layers across dataset scales.
• Figure 5: (a) and (b): comparison of cross-entropy losses between ICL and gradient-based methods. (c) and (d): comparison of GLIM with existing models as the data volume continuously increases.

Theorems & Definitions (15)

• Definition 2.1: Strategic manipulation in SC tasks
• Definition 2.2: Decision rule optimization in SC tasks
• Lemma 1: Forward propagation as implicit gradient descent ahn2023transformers
• Definition 3.1: Strategic Manipulation via ICL (Inner Stage)
• Definition 3.2: Decision Rule Optimization via ICL (Outer Stage)
• Proposition 1: ICL Implements the Gradient-free Strategic Manipulation.
• Remark 1
• Remark 2: Linear Derivation.
• Remark 3
• Proposition 2: ICL Implements Gradient-free Decision Rule Update