Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curvature Tuning: Provable Training-free Model Steering From a Single Parameter

Leyang Hu, Matteo Gamba, Randall Balestriero

TL;DR

Curvature Tuning (CT) introduces a training-free mechanism to steer pretrained models by injecting a single hyperparameter $\beta$ into activation functions, thereby controlling the curvature of decision boundaries without weight updates. CT can be viewed as projection of a ReLU network onto a space of smooth functions, with S-CT as a steering variant and T-CT as a parameter-efficient finetuning method. Empirically, CT improves generalization across 12 downstream datasets on CNN backbones and enhances robustness on RobustBench, while T-CT achieves comparable gains to LoRA with far fewer parameters; CT also shows promise on vision transformers, albeit with weaker gains. The work provides theoretical grounding via the Sobolev projection view and practical techniques for implementing CTU activations, offering a principled complement to weight-based finetuning.

Abstract

The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters. In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model's decision boundary by injecting a single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects a model onto a space of smooth functions-thereby complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. Empirically, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59%/8.34% over linear probing and 4.64%/1.70% over LoRA across 12 datasets, and improves robust accuracy on the $\ell_\infty$ benchmark from RobustBench by 1032.64%/1494.46%. Our code is available at https://github.com/Leon-Leyang/curvature-tuning.

Curvature Tuning: Provable Training-free Model Steering From a Single Parameter

TL;DR

Curvature Tuning (CT) introduces a training-free mechanism to steer pretrained models by injecting a single hyperparameter into activation functions, thereby controlling the curvature of decision boundaries without weight updates. CT can be viewed as projection of a ReLU network onto a space of smooth functions, with S-CT as a steering variant and T-CT as a parameter-efficient finetuning method. Empirically, CT improves generalization across 12 downstream datasets on CNN backbones and enhances robustness on RobustBench, while T-CT achieves comparable gains to LoRA with far fewer parameters; CT also shows promise on vision transformers, albeit with weaker gains. The work provides theoretical grounding via the Sobolev projection view and practical techniques for implementing CTU activations, offering a principled complement to weight-based finetuning.

Abstract

The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters. In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model's decision boundary by injecting a single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects a model onto a space of smooth functions-thereby complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. Empirically, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59%/8.34% over linear probing and 4.64%/1.70% over LoRA across 12 datasets, and improves robust accuracy on the benchmark from RobustBench by 1032.64%/1494.46%. Our code is available at https://github.com/Leon-Leyang/curvature-tuning.

Paper Structure

This paper contains 27 sections, 4 theorems, 27 equations, 6 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Current finetuning techniques and limitations
  4. The spline formulation of deep networks
  5. Curvature Tuning (CT): a provable method for model steering
  6. The $\beta$-Vector-Quantization (VQ) inference framework
  7. Implementation of CT
  8. Curvature Tuning operates as a projection
  9. Enhancing model generalization and robustness with CT
  10. Improving generalization on downstream datasets with S-CT
  11. T-CT is comparable to LoRA with fewer parameters
  12. Improving model robustness through CT's implicit bias
  13. CT shows promise on transformers
  14. Conclusion
  15. Spline theory
  16. ...and 12 more sections

Key Result

Theorem 3.1

For a ReLU network $f: \mathbb{R}^D \to \mathbb{R}$ with parameter $\mathbf{W}$ (collecting all weights and biases), for fixed $c \in {[}0, 1{]}$ and $\beta \in {[}0,1)$, replacing every instance of ReLU with a CTU (eq:CT with hyperparameters $\beta,c$ is equivalent to projecting $f$ to a smooth fun

Figures (6)

  • Figure 1: Illustration of Curvature Tuning (CT) on classification (top) and regression (bottom) tasks. The pretrained model for classification is a 3-layer MLP with hidden width 20 trained for 2000 steps; for regression, it is a 9-layer MLP with hidden width 64 trained for 20000 steps. CT steers a pretrained model by replacing ReLUs with a $\beta$-parameterized activation function and tuning $\beta$ from 1 to 0, effectively modulating the model’s decision boundary curvature.
  • Figure 2: Toy example illustrating how modulating model's activation functions steers decision boundaries curvature. The model is a 2-layer MLP with hidden width 7; (a) baseline pretrained for 4000 steps, then finetuned for another 4000 steps using (b) LoRA ($r=1$, $\alpha=1$) and (c) Trainable CT. CT achieves near-optimal approximation by smoothing the decision boundary of the pretrained model, whereas LoRA only operates on the model parameters, without changing the model's underlying geometry.
  • Figure S1: Visualization of nonlinearity smoothing through region assignment smoothing, max smoothing, and their combination. For a ReLU network, the combined approach mitigates the opposing biases introduced by the individual methods.
  • Figure S2: Validation accuracy (%) of S-CT during the $\beta$ search, averaged over three runs. The accuracy curve varies smoothly and typically peaks in the middle of the $\beta$ range.
  • Figure S3: Common distributions of $\beta$ (top) and $c$ (bottom) in T-CT across ResNet-18/50/152, averaged over three runs (OCTMNIST shown as a representative dataset). Both $\beta$ and $c$ consistently exhibit sharp U-shaped distributions that appear similar across all models.
  • ...and 1 more figures

Theorems & Definitions (6)

  • Theorem 3.1: Informal
  • Theorem A.1
  • Theorem C.1
  • Lemma C.2
  • proof
  • proof