Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions

Heiner Kremer, Jia-Jie Zhu, Krikamol Muandet, Bernhard Schölkopf

TL;DR

This paper addresses estimation under conditional moment restrictions (CMR), which impose a continuum of moment constraints and challenge traditional GMM methods. It develops Functional Generalized Empirical Likelihood (FGEL), a dual, regularized GEL framework that treats conditional moments as a functional object in a Hilbert space and yields a saddle-point estimator. The authors establish consistency, $\sqrt{n}$-rate asymptotics, and efficiency for conditional moment restrictions, and show how functional FGEL translates to CMR via expressive instrument spaces such as RKHS. They provide kernel FGEL and Neural FGEL implementations, with empirical results on linear regression with heteroskedastic noise and instrumental variable regression that achieve state-of-the-art performance on CMR tasks. The work integrates GEL with modern machine learning models, offering a flexible, scalable approach to robust estimation in causal inference and econometrics with continuum moment restrictions.

Abstract

Important problems in causal inference, economics, and, more generally, robust machine learning can be expressed as conditional moment restrictions, but estimation becomes challenging as it requires solving a continuum of unconditional moment restrictions. Previous works addressed this problem by extending the generalized method of moments (GMM) to continuum moment restrictions. In contrast, generalized empirical likelihood (GEL) provides a more general framework and has been shown to enjoy favorable small-sample properties compared to GMM-based estimators. To benefit from recent developments in machine learning, we provide a functional reformulation of GEL in which arbitrary models can be leveraged. Motivated by a dual formulation of the resulting infinite dimensional optimization problem, we devise a practical method and explore its asymptotic properties. Finally, we provide kernel- and neural network-based implementations of the estimator, which achieve state-of-the-art empirical performance on two conditional moment restriction problems.

Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions

TL;DR

This paper addresses estimation under conditional moment restrictions (CMR), which impose a continuum of moment constraints and challenge traditional GMM methods. It develops Functional Generalized Empirical Likelihood (FGEL), a dual, regularized GEL framework that treats conditional moments as a functional object in a Hilbert space and yields a saddle-point estimator. The authors establish consistency, -rate asymptotics, and efficiency for conditional moment restrictions, and show how functional FGEL translates to CMR via expressive instrument spaces such as RKHS. They provide kernel FGEL and Neural FGEL implementations, with empirical results on linear regression with heteroskedastic noise and instrumental variable regression that achieve state-of-the-art performance on CMR tasks. The work integrates GEL with modern machine learning models, offering a flexible, scalable approach to robust estimation in causal inference and econometrics with continuum moment restrictions.

Abstract

Important problems in causal inference, economics, and, more generally, robust machine learning can be expressed as conditional moment restrictions, but estimation becomes challenging as it requires solving a continuum of unconditional moment restrictions. Previous works addressed this problem by extending the generalized method of moments (GMM) to continuum moment restrictions. In contrast, generalized empirical likelihood (GEL) provides a more general framework and has been shown to enjoy favorable small-sample properties compared to GMM-based estimators. To benefit from recent developments in machine learning, we provide a functional reformulation of GEL in which arbitrary models can be leveraged. Motivated by a dual formulation of the resulting infinite dimensional optimization problem, we devise a practical method and explore its asymptotic properties. Finally, we provide kernel- and neural network-based implementations of the estimator, which achieve state-of-the-art empirical performance on two conditional moment restriction problems.
Paper Structure (41 sections, 23 theorems, 87 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 41 sections, 23 theorems, 87 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Learning with Moment Restrictions
  4. Generalized Method of Moments (GMM)
  5. Generalized Empirical Likelihood (GEL)
  6. Functional Generalized Empirical Likelihood
  7. Our Method
  8. Asymptotic Properties
  9. From Functional to Conditional Moment Restrictions
  10. Kernel FGEL
  11. Reproducing kernel Hilbert spaces
  12. Neural FGEL
  13. Other Instrument Function Classes
  14. Choice of Divergence Function
  15. Experiments
  16. ...and 26 more sections

Key Result

Theorem 3.1

Let $\varphi^\ast(v) = \sup_{p \in \mathbb{R}^n} \langle v, p \rangle - \varphi(p)$ denote the Legendre-Fenchel conjugate function of a strongly convex function $\varphi$. Then the problem admits the dual form and strong duality holds between these formulations. Moreover, the unique minimizer of the primal problem is given by where $\hat{h}$, $\hat{\mu}$ are any solutions of the dual problem. M

Figures (2)

  • Figure 1: Estimation error over sample size for the heteroskedastic regression experiment. Lines and shaded regions represent the MSE of the estimated parameters and the standard error averaged over $70$ runs respectively.
  • Figure 2: Comparison of different divergence functions. Lines and shaded regions represent the MSE of the estimated parameters and the standard error averaged over $70$ runs respectively.

Theorems & Definitions (42)

  • Theorem 3.1
  • Remark 3.2
  • Definition 3.3
  • Remark 3.4
  • Remark 3.5
  • Theorem 3.6: Consistency
  • Theorem 3.7: Asymptotic normality
  • Theorem 3.8: Consistency
  • Theorem 3.9: Asymptotic Normality
  • Corollary 3.10: Efficiency
  • ...and 32 more