Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation

TL;DR

Tempora addresses the gap in evaluating test-time adaptation (TTA) under real-world latency constraints. It introduces three time-contingent utility metrics—Discrete, Continuous, and Amortised—that explicitly couple accuracy with latency penalties, enabling evaluation of TTA methods under deployment-relevant timing budgets. Across 240 temporal evaluations on ImageNet-C with seven TTA methods, it reveals rank instability: offline winners frequently underperform when timing is constrained, and no single method dominates across corruption types or pressure levels. The framework highlights three core failure modes and proposes design targets for deployable adaptation, providing a practical lens for method selection and future development in latency-sensitive applications.

Abstract

Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples. This flexibility suits real deployments, yet conventional evaluations unrealistically assume unbounded processing time, overlooking the accuracy-latency trade-off. As ML increasingly underpins latency-sensitive and user-facing use-cases, temporal pressure constrains the viability of adaptable inference; predictions arriving too late to act on are futile. We introduce Tempora, a framework for evaluating TTA under this pressure. It consists of temporal scenarios that model deployment constraints, evaluation protocols that operationalise measurement, and time-contingent utility metrics that quantify the accuracy-latency trade-off. We instantiate the framework with three such metrics: (1) discrete utility for asynchronous streams with hard deadlines, (2) continuous utility for interactive settings where value decays with latency, and (3) amortised utility for budget-constrained deployments. Applying Tempora to seven TTA methods on ImageNet-C across 240 temporal evaluations reveals rank instability: conventional rankings do not predict rankings under temporal pressure; ETA, a state-of-the-art method in the conventional setting, falls short in 41.2% of evaluations. The highest-utility method varies with corruption type and temporal pressure, with no clear winner. By enabling systematic evaluation across diverse temporal constraints for the first time, Tempora reveals when and why rankings invert, offering practitioners a lens for method selection and researchers a target for deployable adaptation.

Figures (4)

• Figure 1: Rank instability across temporal evaluations. Cells show the highest-utility method among seven TTA methods tested on ImageNet-C (ResNet-50). Offline: No time constraints. Temporal: ❶ Discrete (hard deadlines), ❷ Continuous (latency penalties), and ❸ Amortised (budgeted overhead); formal definitions in §3. Rows reveal rank instability: the best method changes under temporal pressure. Similarly, columns show no consistent winner within any scenario. Aggregate benchmarks obscure these corruption-level dynamics. Extended results appear in Appendix \ref{['appendix:extended-results']}.
• Figure 2: Evaluation protocols for measuring time-contingent utility. Each panel depicts the interaction between an input stream, a single model pipeline, and predictions over time $t$. Processing batch $\mathbf{x}_i$ to emit prediction $\mathbf{y}_i$ incurs baseline latency (grey), intrinsic overhead (green) that delays emission, and extrinsic overhead (purple) that stalls the pipeline before the next batch can begin; methods vary in their overhead profile. Offline: The stream waits for adaptation to complete before releasing the next batch; utility is the mean accuracy. ❶ Discrete: Batches arrive asynchronously at fixed intervals; those arriving while the pipeline is occupied are skipped, so utility scales with availability $\alpha$. ❷ Continuous: The stream releases $\mathbf{x}_{i+1}$ upon receiving $\mathbf{y}_i$ (greedy user); response delay incurs hyperbolic penalty $\kappa$. ❸ Amortised: Follows offline until overhead budget is exhausted, then frozen inference only; utility is a weighted combination over both phases.
• Figure 3: Rank instability across temporal scenarios and corruption types. Rows show discrete (utilisation $\rho$), continuous (threshold $T$), and amortised (budget $B$) evaluations; columns show one corruption from each category, spanning baseline accuracies from 2.6% to 59.1%. Green regions mark the best method; grey regions mark methods worse than standard inference. At relaxed thresholds, rankings converge towards offline; under pressure, instability increases. No method dominates. Best viewed in colour.
• Figure 4: Comparison of discrete evaluation protocols.Top:pmlr-v235-alfarra24a's pmlr-v235-alfarra24a dual-model variant uses a fallback model pipeline that serves predictions for batches arriving while the primary model adapts; the dashed lines indicate these batches redirected to the second pipeline. This model may be periodically updated with the adapted state. The setup assumes zero-cost parallel model execution. Middle: Their single-model variant skips missed batches and forces the pipeline to idle until the next arrival. Bottom: Our buffered protocol retains the most recent batch, serving it when the pipeline is free. This avoids fallback assumptions and forced idling while exposing availability as the cost of slow adaptation. The legend largely remains the same as in Figure \ref{['fig:tempora-metrics']}, with the inclusion of pipeline idling (yellow).