Uncertainty quantification in automated valuation models with spatially weighted conformal prediction

TL;DR

The paper tackles calibrated uncertainty quantification for automated valuation models (AVMs) in real estate, addressing spatial heterogeneity that challenges standard conformal prediction (CP). It proposes and evaluates spatially weighted CP variants, including Mondrian-style and local weighting, across multiple non-conformity measures (Standard, Normalized 1/2, and conformalized quantile regression, CQR). Through simulations and an Oslo housing dataset, the study shows that spatial weighting yields more uniformly calibrated coverage across regions, with CQR often delivering the most efficient and robust intervals, while local calibration helps when non-conformity scores miss spatial structure. The findings have practical implications for risk management and decision-making in real estate markets, offering actionable guidance on which CP variants to deploy under spatially varying uncertainty. All mathematical notation in the CP framework is formalized, including $C_{1-\alpha}$, $\hat{q}_{1-\alpha}$, and weighting schemes such as $w_i=\exp(-d_{i,N+1}^2/\eta)$.

Abstract

Non-parametric machine learning models, such as random forests and gradient boosted trees, are frequently used to estimate house prices due to their predictive accuracy, but a main drawback of such methods is their limited ability to quantify prediction uncertainty. Conformal prediction (CP) is a model-agnostic framework for constructing confidence sets around predictions of machine learning models with minimal assumptions. However, due to the spatial dependencies observed in house prices, direct application of CP leads to confidence sets that are not calibrated everywhere, i.e., the confidence sets will be too large in certain geographical regions and too small in others. We survey various approaches to adjust the CP confidence set to account for this and demonstrate their performance on a data set from the housing market in Oslo, Norway. Our findings indicate that calibrating the confidence sets on a spatially weighted version of the non-conformity scores makes the coverage more consistently calibrated across geographical regions. We also perform a simulation study on synthetically generated sale prices to empirically explore the performance of CP on housing market data under idealized conditions with known data-generating mechanisms.

Figures (8)

• Figure 1: Visualizations of the Oslo $2016-2017$ data set. Left: Sale price plotted against the size in $m^2$. Right: Mean price per in million NOK per city district. The white city district has no observations.
• Figure 2: The marginal performance of CP sets at $1-\alpha = 0.9$ on synthetic data set. These results are averaged over $100$ simulations, each time with a new random split into training, calibration, and testing. Left: Empirical coverage gap from the desired $1-\alpha$. Right: Size of the CP set as a percentage of the (synthetic) sale price.
• Figure 3: Mean Absolute Coverage Gap across geographical subsets in the data. Left: Evaluating with the city districts in Oslo as subsets. Right: Evaluating with a $20 \times 20$ grid of the spatial domain as subsets. We display the plots with individual $y$ axes, as the two plots are evaluated on a different spatial granularity.
• Figure 4: The performance of the CP sets at $1-\alpha = 0.9$ on the real data set. Left: Empirical coverage gap from the desired $1-\alpha$. Right: Size of the CP set as a percentage of the actual sale price.
• Figure 5: Mean Absolute Coverage Gap (MACG) across geographical subsets. Left: Evaluating with the city districts in Oslo as subsets. Right: Evaluating with a $20 \times 20$ grid of the spatial domain as subsets. We display the plots with different scales on $y$ axis, as the two plots are evaluated on a different spatial granularity.