Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SteeringTTA: Guiding Diffusion Trajectories for Robust Test-Time-Adaptation

Jihyun Yu, Yoojin Oh, Wonho Bae, Mingyu Kim, Junhyug Noh

TL;DR

SteeringTTA introduces an inference-only diffusion-based test-time adaptation framework that uses Feynman-Kac steering with a pseudo-label reward to guide multiple diffusion trajectories. By maintaining K parallel particles and employing ESS-based resampling, a flexible proposal kernel, and a low-pass semantic filter, it balances exploration of plausible corruptions with convergence toward the correct class. The method demonstrates consistent improvements over diffusion-based baselines on ImageNet-C without updating model parameters or relying on source data, with GT (ground-truth reward) providing an upper bound. This approach offers a practical, non-gradient-based mechanism to enhance robustness under severe, low-frequency corruptions. Potential extensions include adaptive resampling strategies and broader applications beyond image classification.

Abstract

Test-time adaptation (TTA) aims to correct performance degradation of deep models under distribution shifts by updating models or inputs using unlabeled test data. Input-only diffusion-based TTA methods improve robustness for classification to corruptions but rely on gradient guidance, limiting exploration and generalization across distortion types. We propose SteeringTTA, an inference-only framework that adapts Feynman-Kac steering to guide diffusion-based input adaptation for classification with rewards driven by pseudo-label. SteeringTTA maintains multiple particle trajectories, steered by a combination of cumulative top-K probabilities and an entropy schedule, to balance exploration and confidence. On ImageNet-C, SteeringTTA consistently outperforms the baseline without any model updates or source data.

SteeringTTA: Guiding Diffusion Trajectories for Robust Test-Time-Adaptation

TL;DR

SteeringTTA introduces an inference-only diffusion-based test-time adaptation framework that uses Feynman-Kac steering with a pseudo-label reward to guide multiple diffusion trajectories. By maintaining K parallel particles and employing ESS-based resampling, a flexible proposal kernel, and a low-pass semantic filter, it balances exploration of plausible corruptions with convergence toward the correct class. The method demonstrates consistent improvements over diffusion-based baselines on ImageNet-C without updating model parameters or relying on source data, with GT (ground-truth reward) providing an upper bound. This approach offers a practical, non-gradient-based mechanism to enhance robustness under severe, low-frequency corruptions. Potential extensions include adaptive resampling strategies and broader applications beyond image classification.

Abstract

Test-time adaptation (TTA) aims to correct performance degradation of deep models under distribution shifts by updating models or inputs using unlabeled test data. Input-only diffusion-based TTA methods improve robustness for classification to corruptions but rely on gradient guidance, limiting exploration and generalization across distortion types. We propose SteeringTTA, an inference-only framework that adapts Feynman-Kac steering to guide diffusion-based input adaptation for classification with rewards driven by pseudo-label. SteeringTTA maintains multiple particle trajectories, steered by a combination of cumulative top-K probabilities and an entropy schedule, to balance exploration and confidence. On ImageNet-C, SteeringTTA consistently outperforms the baseline without any model updates or source data.
Paper Structure (40 sections, 9 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 40 sections, 9 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Test‐Time Adaptation.
  4. Diffusion‐Based Input Adaptation.
  5. SMC and FK Steering.
  6. Our Approach
  7. Preliminaries
  8. Diffusion Models.
  9. Sequential Monte Carlo (SMC).
  10. Feynman-Kac (FK) Steering.
  11. Generalized FK steering for TTA
  12. Test-time Reward
  13. Candidate set construction.
  14. Reward formulation.
  15. Annealing schedule.
  16. ...and 25 more sections

Figures (3)

  • Figure 1: (a) Our motivation: from left to right, the original corrupted image, the DDA‐denoised result, and our method's reconstruction, with corresponding ResNet‐50 predictions. (b) Pipeline overview: at each diffusion step the image is updated, the classifier output is used to compute a reward, and the update is steered to maximize this reward.
  • Figure D.1: Qualitative results comparing the original corrupted image, DDA, GT-based SteeringTTA and ours.
  • Figure D.2: From top to bottom, the adapted image with Grad-DDA ($scale=10$) and our method's with same rewards. Some unreliable high-frequency artifacts appear in Grad-DDA which are not in ours.