Context-Aware Inference via Performance Forecasting in Decentralized Learning Networks

TL;DR

The paper tackles the slow adaptation of weightings in decentralized learning by introducing context-aware performance forecasting that predicts per-epoch model performance. Using Allora-inspired networks, forecasting workers predict losses or derived regrets (and regret $z$-scores), which are converted through a sigmoid gate to forecast-implied weights for combining inferences. Across synthetic benchmarks and live Allora data, forecasts based on regret and regret $z$-scores consistently outperform naive, history-based network inferences, with per-inferer models typically offering the strongest context awareness. The findings highlight the importance of target choice, feature design, and training horizon, and suggest that performance forecasting can be a practical, generalizable approach for predictive weighting in dynamic ensembles. This work provides a concrete framework and open-source baseline for deploying context-aware forecasting in decentralized prediction tasks and potentially broader model-combination problems.

Abstract

In decentralized learning networks, predictions from many participants are combined to generate a network inference. While many studies have demonstrated performance benefits of combining multiple model predictions, existing strategies using linear pooling methods (ranging from simple averaging to dynamic weight updates) face a key limitation. Dynamic prediction combinations that rely on historical performance to update weights are necessarily reactive. Due to the need to average over a reasonable number of epochs (with moving averages or exponential weighting), they tend to be slow to adjust to changing circumstances (phase or regime changes). In this work, we develop a model that uses machine learning to forecast the performance of predictions by models at each epoch in a time series. This enables `context-awareness' by assigning higher weight to models that are likely to be more accurate at a given time. We show that adding a performance forecasting worker in a decentralized learning network, following a design similar to the Allora network, can improve the accuracy of network inferences. Specifically, we find forecasting models that predict regret (performance relative to the network inference) or regret z-score (performance relative to other workers) show greater improvement than models predicting losses, which often do not outperform the naive network inference (historically weighted average of all inferences). Through a series of optimization tests, we show that the performance of the forecasting model can be sensitive to choices in the feature set and number of training epochs. These properties may depend on the exact problem and should be tailored to each domain. Although initially designed for a decentralized learning network, using performance forecasting for prediction combination may be useful in any situation where predictive rather than reactive model weighting is needed.

Figures (12)

• Figure 1: True and predicted regret values for two 'inferers' in a simple periodic (sinusoidal) outperformance test. In the left panel the sine function has an amplitude of 1 and period of 10 epochs, while in the right panel the sine function has an amplitude of 1.5 and period of 17 epochs. A random noise term (uniformly sampled $\in[-1,1]$) is added at each epoch for both inferers. In both cases, the default per-inferer forecasting model reasonably identifies both the amplitudes and periods of the underlying sinusoidal evolution for each worker.
• Figure 2: Comparison of true and predicted regrets in the sinusoidal evolution test for the global (left panel) and per-inferer (right panel) forecasting models. Small data points show individual values for each inferer at each epoch and large open squares show the median of true and predicted regrets for each inferer. Coloured dashed lines show linear fits for each inferer, using Huber regression to minimize effects of outliers. The black dashed line shows the ideal one-to-one relation. Though the overall performance is similar (as indicated by the RMSE in each panel), the global model shows some confusion in predicted regrets between the two 'outperforming' workers (allo0 and allo1, e.g. at values $< 0$), while the per-inferer model clearly distinguishes their predictions (clear bands at $-1$ and $-1.5$, respectively).
• Figure 3: Fixed interval periodic outperformance test ($+1$ in regret every 10 epochs), with (left) and without (right) autocorrelation for the per-inferer forecasting model. This test shows that autocorrelation is crucial to identify the outperformance epochs.
• Figure 4: Top: Ground truth for the contextual outperformance test, with shaded regions highlighting the periods with a non-zero drift parameter (blue: downward drift; red: upward drift). Bottom: Comparison of individual inferences (grey points), naive network inference (red dashed line), and forecast-implied inference (solid orange line) with the ground truth (solid blue line). The forecast-implied inference was generated using forecasted $z$-scores from the default per-inferer forecaster model. The mean log loss of the naive network and forecast-implied inferences are indicated in the legend, with the forecast-implied inference ($\log{L}=0.57$) clearly outperforming the naive network inference ($\log{L}=1.09$).
• Figure 5: 'Context awareness' of forecasting models in the contextual outperformance test. Panels show the true versus predicted properties for forecasters predicting log losses (left), regrets (middle) and regret $z$-scores (right). Point and line styles are as in \ref{['fig:sine_pred']}. The legend is split over the three panels; all inferers are present in each panel.