Evolution and The Knightian Blindspot of Machine Learning

TL;DR

Knightian uncertainty represents a qualitatively unknown future that current ML/RL formalisms largely dodge. By contrasting RL with biological evolution, the paper shows how open-ended diversification, environmental interaction, and long-horizon falsification enable robustness to unforeseen challenges, while MDP-based RL exhibits time-blindness, narrow risk concepts, and static deployment assumptions. It argues that enriching ML with diversification-and-filter strategies, artificial life, and open-ended search—plus revising RL formalism to better accommodate KU—could yield truly open-world robustness and advance toward AGI. Foundation models and RLHF may appear robust due to scale and interpolation, but KU-sensitive failures persist, underscoring the need for KU-aware theory and algorithms with practical impact for AI safety and open-world deployment.

Abstract

This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to Knightian uncertainty (KU) in economics, i.e. uncertainty that cannot be quantified, which is excluded from consideration in ML's key formalisms. This paper aims to identify this blind spot, argue its importance, and catalyze research into addressing it, which we believe is necessary to create truly robust open-world AI. To help illuminate the blind spot, we contrast one area of ML, reinforcement learning (RL), with the process of biological evolution. Despite staggering ongoing progress, RL still struggles in open-world situations, often failing under unforeseen situations. For example, the idea of zero-shot transferring a self-driving car policy trained only in the US to the UK currently seems exceedingly ambitious. In dramatic contrast, biological evolution routinely produces agents that thrive within an open world, sometimes even to situations that are remarkably out-of-distribution (e.g. invasive species; or humans, who do undertake such zero-shot international driving). Interestingly, evolution achieves such robustness without explicit theory, formalisms, or mathematical gradients. We explore the assumptions underlying RL's typical formalisms, showing how they limit RL's engagement with the unknown unknowns characteristic of an ever-changing complex world. Further, we identify mechanisms through which evolutionary processes foster robustness to novel and unpredictable challenges, and discuss potential pathways to algorithmically embody them. The conclusion is that the intriguing remaining fragility of ML may result from blind spots in its formalisms, and that significant gains may result from direct confrontation with the challenge of KU.

Figures (5)

• Figure 1: Interlocking principles enabling evolution's robustness to Knightian Uncertainty. (a) Evolution happens within a search space that is open-ended enough such that a vast array of complex adaptations can be encoded, e.g. the human brain, multicellularity, developmental systems as a whole, and photosynthesis. (b) Diversification pressure in evolution continually creates new behaviors and adaptations from the set of open-ended possibilities, which implicitly can be seen as bets about how the organism and its lineage can persist into the future. (c) Because organisms form part of the environment of other organisms, novel behaviors and adaptations in one lineage create novel unforeseen situations for other organisms as an externality, e.g. the high branches of a tree provide a novel situation a giraffe can exploit. (d) Organisms unable to persist across the uncertainty created by other organisms are filtered away, in effect invalidating their bets about how to persist through KU; the image shows a coelacanth, a fish that has persisted for 400 million years. In concert, these factors can be seen as a form of open-ended generation and falsification of bets about how to deal with KU. We believe there may be ways to adapt these principles to ML research (see discussion in Section \ref{['sec:discussion']}).
• Figure 2: Two Strategies for Dealing with an Open World. This figure describes two possible strategies for coping with an open changing world. In (a) diversify-and-filter, a process continually refreshes and adapts its diverse hypotheses about how to persist through the open-ended future. Such hypotheses are filtered through empirical success at tackling later unanticipated problems. Evolution, market competition, and science can be seen to largely operate through this paradigm. There is no explicit formalism, although robustness implicitly relies on the Lindy effect taleb2014antifragile, i.e. an adaptable solution long-tested by time is more likely than an untested one to persist yet longer. In (b) anticipate-and-train, diverse problems are first collected, and augmented through human anticipation about what novel situations might later arise. Then, a single policy is trained to solve the problems to convergence, and that policy is then deployed into a changing world. Much of current ML adopts this paradigm; although the closed-world formalism adopted in training mismatches the open world it is deployed into, the hope is that generalization will enable sufficient robustness to unforeseen challenges. One conclusion is that nothing precludes machine learning from more deeply integrating diversify-and-filter approaches into its methods lehman2010revisingkumar2020onejaderberg2017populationlee2023diversify. Another conclusion is that diversify-and-filter leverages the temporal structure of when novel problems arise, and forces agents to directly grapple with the issue of KU (if they do not, they are discarded).
• Figure 3: Optimizing for known unknowns can exacerbate risk from Knightian uncertainty. An optimization formalism that makes closed-world assumptions will indeed improve an agent's performance on the situations an experimenter anticipates. However, if such a closed-world optimizer aggressively trains an open-world agent, the agent may perversely become more brittle to Knightian uncertainty, as it is incentivized to internalize the closed-world assumptions as true.
• Figure 4: Neural network generalization is not a general cure for Knightian Uncertainty. Imagine as part of a larger reinforcement learning policy, an agent decides whether to eat certain mushrooms, which can either be deadly or edible, and can be separated through features learned in training that correspond to the cap size of the mushroom and its thickness. (a) In a closed world, it is safe to assume that the distribution of mushrooms encountered during training (red $\times$'s and green $+$'s) reflects that encountered during testing, and the (b) NN decision boundary on whether to eat or not eat the mushroom learned through training will likely reflect this assumption. However, in an open world, not all mushroom varieties are known, the policy might be deployed in a slightly different ecosystem, or a new variety of mushroom might evolve or be bred. If encountering the unanticipated mushroom (question mark symbol) at the center of (a), it is likely rational for an open-world agent to forgo eating it, given its novelty and the risk of death. The claim is that simple generalization from what is known does not address Knightian uncertainty.
• Figure 5: Typical metalearning setups do not incentivize learning how to solve unforeseen tasks. This figure offers a caricature of optimal behavior under a typical meta-RL formalism, where an agent is trained across a fixed distribution of problems; this setup is similar to e.g. duan2016rl. In meta-RL, it is common for an agent to be exposed many times to training tasks covering all major necessary task-relevant skills. Thus the agent is incentivized to learn in training all qualitative skills needed to solve the tasks; after many iterations of training, there need be no significant remaining surprise for the agent when solving new tasks drawn from an IID test distribution (which is the formal goal of the algorithm). At completion of training, an optimal agent's behavior is sketched as: (1) it encounters a task drawn from the IID test distribution, which is ambiguous (as this characterizes the need for metalearning); (2) the agent takes actions that optimally disambiguate the sampled task; and (3) having identified the task, which it has encountered many similar variants of before in training, the agent executes its previously-learned optimal solution. In practice, optimal behavior will entail mixing steps (2) and (3) together, but nowhere in this process does optimality under the formalism require the generalized ability to learn how to learn. The conclusion is that if then deployed into a changing world where it encounters an unknown unknown, the agent may struggle to handle it gracefully.