Agnostic Language Identification and Generation
Mikael Møller Høgsgaard, Chirag Pabbaraju
TL;DR
The paper advances the study of language identification and generation in an agnostic setting where the data distribution ${\mathcal D}$ may not align with any language in a reference collection ${\mathcal C}$. It defines the agnostic objectives ${\rm IdErr}$ for identification and ${\rm GenErr}$ for generation, and characterizes when fast (exponential) statistical rates are achievable. For agnostic identification, exponential rates are possible if the infimum over ${\mathcal C}$ is attained by some language, but can be arbitrarily slow otherwise; for agnostic generation, a general lower bound shows intractability without structural assumptions, while a finite ${\mathcal C}$ and a support containment condition yield matching exponential rates, with a proven lower bound that demonstrates tightness. The results illuminate the delicate dependency of learnability on the geometry between the distribution support and the reference languages, and raise questions about extending exponential rates to broader countable collections and alternative objective formulations.
Abstract
Recent works on language identification and generation have established tight statistical rates at which these tasks can be achieved. These works typically operate under a strong realizability assumption: that the input data is drawn from an unknown distribution necessarily supported on some language in a given collection. In this work, we relax this assumption of realizability entirely, and impose no restrictions on the distribution of the input data. We propose objectives to study both language identification and generation in this more general "agnostic" setup. Across both problems, we obtain novel interesting characterizations and nearly tight rates.