Order-Aware Test-Time Adaptation: Leveraging Temporal Dynamics for Robust Streaming Inference

TL;DR

OATTA addresses distribution shifts in streaming inference by explicitly modeling temporal dynamics through a learned transition matrix that regularizes predictions via recursive Bayesian updates. It operates as a lightweight, gradient-free wrapper that can be attached to existing TTA baselines, and it optionally uses a likelihood-ratio gate to safely revert to the base predictor when temporal structure is weak. The key contributions are the online transition estimation mechanism, the temporal prior fusion, and the LLR gate that balances temporal evidence with order-agnostic baselines. Empirically, OATTA improves robustness across image, sensor, and language domains, with gains up to $6.35\%$ on highly temporal streams and additive improvements when combined with baselines like TTAug, demonstrating the practical value of leveraging temporal dynamics in streaming inference.

Abstract

Test-Time Adaptation (TTA) enables pre-trained models to adjust to distribution shift by learning from unlabeled test-time streams. However, existing methods typically treat these streams as independent samples, overlooking the supervisory signal inherent in temporal dynamics. To address this, we introduce Order-Aware Test-Time Adaptation (OATTA). We formulate test-time adaptation as a gradient-free recursive Bayesian estimation task, using a learned dynamic transition matrix as a temporal prior to refine the base model's predictions. To ensure safety in weakly structured streams, we introduce a likelihood-ratio gate (LLR) that reverts to the base predictor when temporal evidence is absent. OATTA is a lightweight, model-agnostic module that incurs negligible computational overhead. Extensive experiments across image classification, wearable and physiological signal analysis, and language sentiment analysis demonstrate its universality; OATTA consistently boosts established baselines, improving accuracy by up to 6.35%. Our findings establish that modeling temporal dynamics provides a critical, orthogonal signal beyond standard order-agnostic TTA approaches.

Figures (4)

• Figure 1: Illustration of Order-Aware Test-Time Adaptation (OATTA) using a human activity recognition example. Unlike order-agnostic TTA, OATTA leverages temporal structure through recursive Bayesian estimation. (1) The problem (sensor ambiguity): As shown in the "Sensor Noise" bar chart, the backbone's raw prediction ($q_t$) can be ambiguous (e.g., confusing "Walk" with "Run"). We convert $q_t$ to a likelihood ($\mathcal{L}_t$) using a uniform prior ($\mathcal{U}$), since source statistics are unavailable. (2) The solution (temporal logic): This likelihood is fused with a temporal prior ($\pi_t$), obtained by projecting the previous posterior ($p_{t-1}$) through the dynamic transition matrix ($A$). (3) Result: In this example, although the sensor is uncertain, OATTA promotes "Walk" over "Run." This occurs because $A$ has learned that the transition "Sit $\to$ Run" is physically improbable, effectively using temporal context to filter out the sensor noise.
• Figure 2: Impact of behavioral diversity on adaptation (SENT). We visualize the learned transition dynamics ($p(Pos|Pos)$ vs. $p(Neg|Neg)$) for 150 unique users. The wide dispersion of points demonstrates significant heterogeneity, invalidating the utility of a single, universal transition rule. Color denotes the per-user accuracy change relative to the base model, $\Delta = \mathrm{Acc}(\text{Ours}) - \mathrm{Acc}(\text{Base})$ (blue: positive gain; red: negative gain).
• Figure 3: Adaptation plasticity under regime shift ($\alpha=0.7 \to 0.5$). We track the structural gain on the CIFAR-10 stream. The light blue trace shows the noisy base signal, while the solid line (EMA, span=300) reveals the adaptation trend. At $t=1{,}000$, the stream becomes less predictable; OATTA detects this shift and rapidly down-calibrates its prior strength, stabilizing at a new equilibrium that accurately reflects the reduced temporal signal.
• Figure 4: Base model utility vs. adaptation gain. Correlation analysis on CIFAR-10-C (level 5) evaluated on S2 (sticky) streams. Each point represents one of the 15 corruption types. We observe a strong linear correlation ($R \approx 0.82$), with an empirical break-even point at 48.2% accuracy.