What if Eye...? Computationally Recreating Vision Evolution

TL;DR

This work introduces a computational framework that co-evolves eye morphology, optics, and neural processing in embodied agents to test hypotheses about vision evolution under environmental pressures. By isolating tasks in a What if World and employing nested CMA-ES with inner PPO learning, it demonstrates task-driven bifurcations into compound versus camera-type eyes, the emergence of lens-based optical elements to balance light collection and acuity, and scaling laws linking sensory acuity, neural capacity, and memory. The findings reveal principled trade-offs and design principles that illuminate both natural vision evolution and bio-inspired artificial vision systems, while providing a versatile hypothesis-testing platform. The approach advances understanding of how environmental demands shape sensory hardware and computation, with potential applications in engineered vision systems and AI.

Abstract

Vision systems in nature show remarkable diversity, from simple light-sensitive patches to complex camera eyes with lenses. While natural selection has produced these eyes through countless mutations over millions of years, they represent just one set of realized evolutionary paths. Testing hypotheses about how environmental pressures shaped eye evolution remains challenging since we cannot experimentally isolate individual factors. Computational evolution offers a way to systematically explore alternative trajectories. Here we show how environmental demands drive three fundamental aspects of visual evolution through an artificial evolution framework that co-evolves both physical eye structure and neural processing in embodied agents. First, we demonstrate computational evidence that task specific selection drives bifurcation in eye evolution - orientation tasks like navigation in a maze leads to distributed compound-type eyes while an object discrimination task leads to the emergence of high-acuity camera-type eyes. Second, we reveal how optical innovations like lenses naturally emerge to resolve fundamental tradeoffs between light collection and spatial precision. Third, we uncover systematic scaling laws between visual acuity and neural processing, showing how task complexity drives coordinated evolution of sensory and computational capabilities. Our work introduces a novel paradigm that illuminates evolutionary principles shaping vision by creating targeted single-player games where embodied agents must simultaneously evolve visual systems and learn complex behaviors. Through our unified genetic encoding framework, these embodied agents serve as next-generation hypothesis testing machines while providing a foundation for designing manufacturable bio-inspired vision systems. Website: http://eyes.mit.edu/

Figures (16)

• Figure 1: Computational evolution of embodied artificial intelligence (AI) agents reveals how environmental pressures shaped natural vision evolution. We evolve artificial embodied agents to show how three evolutionary branch points shaped vision evolution. We use our framework to understand (a) how environmental specificity led to distinct eye morphologies, (b) how optical elements emerge when embodied agents evolve to discriminate between objects while accounting for physical trade-offs of light throughput vs. spatial precision in an environment, and (c) how visual task error scales as power law with visual acuity and number of parameters revealing that poor visual acuity creates a fundamental bottleneck that cannot be overcome by simply scaling neural capacity. (d) Our framework mirrors natural selection: an outer loop governs genetic inheritance and selection over evolutionary timescales, while an inner loop enables agents to learn through sensory feedback (lifetime adaptation). This nested structure reflects the Baldwin effect baldwin1896new, where lifetime learning can guide and accelerate evolutionary adaptation. (e) The agent's digital anatomy parallels biological visual systems: from eye morphology and placement, through optical elements and photoreceptors (mimicking retinal organization), to neural processing (analogous to visual cortex). (f) Agents are evolved to solve three distinct visual tasks to probe how environmental pressures shape vision: (i)Navigation: orientation and obstacle avoidance through a maze-like environment; (ii)Detection: object discrimination between a goal object (food) and an adversarial object (poison); (iii)Tracking: identical to Detection, but the objects move. Our results highlight how embodied agents can serve as scientific instruments to understand biological visual intelligence.
• Figure 2: Our genetic encoding enables vision to evolve computationally. Our encoding mirrors the natural separation between sensory and neural development through three gene clusters. Morphological genes determine agent properties relating to spatially sampling the environment such as eye placement and field of view. Optical genes determine agent properties relating to how each eye interacts with incoming light in a physically plausible way such as # photoreceptors, optical elements, pupil size). Neural genes describe the behavior learning capacity of the agent. These independently mutable genes enable the computational exploration of evolutionary pathways that mirror those in natural vision evolution.
• Figure 3: Low- and high-acuity spatial tasks lead to compound and camera eyes, respectively.(a) We initialize a population of agents for two visual tasks (Detection and Navigation) with a single eye with one photoreceptor. We then evolve a population of agents subject to morphological mutations: add photoreceptor, add eye, and adjust placement. In the Navigation task, we first observe an emergence of dispersed vision, where many eyes are employed. By 50 generations, a compound-type eye emerges; that is, a vision system consisting of 10 individual eyes, each with 16 photoreceptors (4 $\times$ 1 resolution), distributed over the entire diameter of the agent. (c) In the Detection task, we initially observe the emergence low-resolution vision. After 50 generations, the population has converged on a morphology consisting of two forward facing, high-resolution camera-type eyes each with 225 photoreceptors (15$\times$15 resolution). (d) Configuration vs generation plots are shown, depicting the evolutionary progression of the mean agent in the population and the task dependence on evolutionary adaptation. The plots show the mean and 95% bootstrapped confidence interval, respectively.
• Figure 4: Computational evolution reveals how lensing resolves a fundamental trade-off in vision. We demonstrate that to achieve maximum fitness in the Detection task, evolution learns to evolve optical structures against two competing objectives: achieving high spatial precision and maximizing light collection. (a) The evolutionary sequence shows five key stages of eye development: (1) open pupil with maximum light collection but poor spatial precision, (2) cup eye and (3) pinhole eye that achieve better spatial precision by reducing pupil size at the cost of light collection, followed by the emergence of (4) unfocused and (5) focused lens-based eyes that maintain spatial precision by evolving optical structures while allowing larger pupils for more light collection. Agent images show the scene as perceived at each stage. (b) Without optics (dark blue), pupil size decreases to improve precision, sacrificing the signal-to-noise ratio (SNR). When lensing is enabled (orange line, generation 30), larger pupils emerge as lenses are evolved maintain precision while increasing light throughput. (c) The Image Quality metric (image sharpness × light throughput) quantifies this trade-off resolution: pinhole eyes (3) plateau at low values due to limited light collection, while lens-based eyes (4,5) achieve higher quality by combining good spatial precision with larger pupils. This mirrors the evolutionary pressure that drove the emergence of biological lenses, which enabled enhanced vision across lighting conditions.
• Figure 5: Task-dependent scaling laws reveal how sensory acuity bounds performance and how temporal memory compensates for neural capacity(a-c) Our experiments reveal visual task-dependent power law scaling between number of parameters and sensory acuity (CPD). This demonstrates that scaling in sensory input is required for embodied tasks to avoid a bottleneck that cannot be overcome by neural scaling alone. (d) A minimum required visual acuity compared to number of parameters for different embodied tasks suggest a hierarchy in the emergent behaviors depending on the task. (e) Temporal processing shows complementary scaling with neural capacity, where increased temporal memory (# Frames) can compensate for reduced neural processing, which is particularly evident in tasks with larger networks. Together, these results quantify how visual intelligence emerges from the interplay between sensory, neural, and temporal capabilities.