Risk reversal for least squares estimators under nested convex constraints

TL;DR

The paper uncovers a surprising failure mode of projection-based estimators: constraining the parameter to a smaller convex set can increase the risk, even when the smaller set contains the truth. Through a concrete two-dimensional example and two asymptotic regimes, it shows risk reversal occurs in the diverging-noise limit due to global geometry, while in the vanishing-noise limit the effect vanishes for interior points and is dominated by boundary geometry at higher orders. It establishes a rigorous worst-case reversal for nested convex polytopes by proving uniform convergence of the finite-noise risk to a limiting polytope-based risk, which can reverse the usual monotonicity under set inclusion. The results are grounded in convex geometry and Gaussian process projections, linking the behavior to tangent cones, statistical dimension, and face-selection mechanisms determined by random noise directions. Overall, tightening a constraint can paradoxically degrade performance in sufficiently noisy settings, motivating a more nuanced view of prior information incorporation in constrained estimation.

Abstract

In constrained stochastic optimization, one naturally expects that imposing a stricter feasible set does not increase the statistical risk of an estimator defined by projection onto that set. In this paper, we show that this intuition can fail even in canonical settings. We study the Gaussian sequence model, a deliberately austere test best, where for a compact, convex set $Θ\subset \mathbb{R}^d$ one observes \[ Y = θ^\star + σZ, \qquad Z \sim N(0, I_d), \] and seeks to estimate an unknown parameter $θ^\star \in Θ$. The natural estimator is the least squares estimator (LSE), which coincides with the Euclidean projection of $Y$ onto $Θ$. We construct an explicit example exhibiting \emph{risk reversal}: for sufficiently large noise, there exist nested compact convex sets $Θ_S \subset Θ_L$ and a parameter $θ^\star \in Θ_S$ such that the LSE constrained to $Θ_S$ has strictly larger risk than the LSE constrained to $Θ_L$. We further show that this phenomenon can persist at the level of worst-case risk, with the supremum risk over the smaller constraint set exceeding that over the larger one. We clarify this behavior by contrasting noise regimes. In the vanishing-noise limit, the risk admits a first-order expansion governed by the statistical dimension of the tangent cone at $θ^\star$, and tighter constraints uniformly reduce risk. In contrast, in the diverging-noise regime, the risk is determined by global geometric interactions between the constraint set and random noise directions. Here, the embedding of $Θ_S$ within $Θ_L$ can reverse the risk ordering. These results reveal a previously unrecognized failure mode of projection-based estimators: in sufficiently noisy settings, tightening a constraint can paradoxically degrade statistical performance.

Figures (5)

• Figure 1: Geometry of the LSE over constraint sets $\Theta_S = \operatorname{conv}(v_1,v_2)$ (left) and $\Theta_L=\operatorname{conv}(v_1,v_2, v_3)$ (right). The shaded regions illustrate the partition of the sample space based on the location of the constrained solution: regions $A_j$ denote observations projected onto vertex $v_j$, while regions $A_{ij}$ denote observations projected onto the line segment connecting $v_i$ and $v_j$.
• Figure 2: Risk difference as a function of noise level $\sigma$. The difference $R_\sigma(\theta^\star; \Theta_S)-R_\sigma(\theta^\star; \Theta_L)$ is plotted against $\sigma$ for varying values of the geometric parameter $c$. Note that for $c \in \{0.2,0.5,0.9 \}$, $\widehat{\theta}_L$ outperforms $\widehat{\theta}_S$ whenever the noise level is sufficiently large.
• Figure 3: Heat map of the risk difference $R_\sigma(\theta^\star; \Theta_S) - R_\sigma(\theta^\star; \Theta_L)$ as a function of the geometric parameter $c$ (horizontal axis) and the noise level $\sigma$ (vertical axis). Risk reversal occurs in a substantial intermediate-to-large noise regime for values of $c \in (0,1)$.
• Figure 4: A schematic illustration of the risk reversal phenomenon. A realization $Y$ is projected onto the larger constraint set, yielding $\widehat{\theta}_{L}$, and onto the smaller set, yielding $\widehat{\theta}_{S}$. In this configuration, the distance from $\theta^\star$ of projecting onto the larger set, $d_L(Y) = \|\Pi_{\Theta_L}(Y) - \theta^\star \|$, is smaller than the corresponding distance of projecting onto the smaller set, $d_S(Y)= \|\Pi_{\Theta_S}(Y) - \theta^\star \|$. The shaded region denotes the set $\mathcal{G} = \{ y: d_L(y) < d_S(y)\}$ of all points for which projecting on to the small set is relatively worse. When the noise level $\sigma$ is sufficiently large, the Gaussian centered at $\theta^\star$ places enough mass on $\mathcal{G}$ for the risk associated with projection onto $\Theta_L$ to be smaller than that associated with projection onto $\Theta_S$.
• Figure 5: Vertex risks $R_\infty(v_1;\Theta_x)$, $R_\infty(v_2;\Theta_x)$, and $R_\infty(v_x;\Theta_x)$, together with their upper envelope $\overline{R}_\infty(\Theta_x)$, as functions of $x$ for $c=0.75$. The envelope has a minimum at $x=0.4290.$ Increasing $x$ tightens the constraint. The non-monotonicity of the envelope implies worst-case risk reversal (see Remark \ref{['rem:worst-case-intuition']}).

Theorems & Definitions (32)

• Example 1.1
• Theorem 1.1
• Theorem 1.2
• Theorem 2.1
• Corollary 2.2
• Theorem 2.3
• Lemma 2.4
• Corollary 2.5
• Lemma 2.6
• Theorem 2.7