Artificial Scientific Discovery

TL;DR

The thesis investigates what it would take for machines to become artificial scientists capable of autonomous discovery and explanation. It introduces Explanatory Learning (EL) to learn interpreters from explanation–observation data, and CRNs to ground explanations with rationalist constraints, demonstrated via Odeen as a testbed. It then reframes cross-modal grounding with ASIF, a training-free method that aligns two frozen encoders using relative representations to yield CLIP-like multimodal models without end-to-end training. Finally, it critically assesses Large Language Models, showing they struggle with symbol interpretation tasks that humans solve, suggesting that future artificial scientists must couple language with robust reasoning, world models, and interactive exploration. Together, these threads chart a path toward machines that discover, explain, and adapt across modalities, with implications for data-centric AI and the design of interpretable scientific agents.

Abstract

Rooted in the explosion of deep learning over the past decade, this thesis spans from AlphaGo to ChatGPT to empirically examine the fundamental concepts needed to realize the vision of an artificial scientist: a machine with the capacity to autonomously generate original research and contribute to the expansion of human knowledge. The investigation begins with Olivaw, an AlphaGo Zero-like agent that discovers Othello knowledge from scratch but is unable to communicate it. This realization leads to the development of the Explanatory Learning (EL) framework, a formalization of the problem faced by a scientist when trying to explain a new phenomenon to their peers. The effective EL prescriptions allow us to crack Zendo, a popular board game simulating the scientific endeavor. This success comes with a fundamental insight: an artificial scientist must develop its own interpretation of the language used to explain its findings, and not rely on a rigid existing interpreter. Questioning the very process of learning an interpreter, we turn our attention to the inner functioning of modern multimodal models. This culminates in a simple idea to build CLIP-like models where interpretation and perception are explicitly disentangled: a cost-effective approach that couples two unimodal models using little multimodal data and no further training. Finally, we discuss what ChatGPT and its siblings are still missing to become artificial scientists, and introduce the Big-Bench Symbol Interpretation Task, a benchmark about interpreting Zendo-like explanations that sees LLMs going no further than random chance while being instead fully solved by humans.

Figures (38)

• Figure 1: Rules of Othello. Othello is a turn-based game where the black and white player try to overcome each other in the final domination of an 8x8 board. a. Players move alternately by placing a new disk in an empty square in order to bracket one or more opponent’s disks between the played disk and another of its own color already on the board. It is possible to capture disks horizontally, vertically, and diagonally. Disks can be captured in one or more directions in a single move, with capture always occurring in a straight line. Only moves capturing at least one disk are allowed. In the absence of moves the player must skip the turn. It is not possible to pass the turn if there is at least one valid move. b. The imprisoned disks change color and become owned by the player who moved. c. When none of the players can move, for instance when the board is full, the player with more disks on the board wins. Here black wins 40-24. d. A game of Othello begins with 4 disks placed in the center of the board in the shape of an X. Black moves first.
• Figure 2: Olivaw's Training process.a. Training loss across generations. The stepwise trend is due to the shifting training window. b. Absolute largest value drop in a game across generations. We show the averages using standard deviation as confidence interval. c. Performance of the i-th generation MCTS agent (continuous line) and number of games played by Olivaw against itself during training in each generation (bar chart). The ELO ratings were computed using the evaluation games and the first match vs the national champion Alessandro Di Mattei.
• Figure 3: Olivaw's performance on OthelloQuest.The score of different generations of Olivaw on OthelloQuest. We report the outcome of the last 50 games played by every version of Olivaw (the initial matches are warm-up games used by the platform to assess the strength of the player and thus are excluded). Every agent played anonymously using 400 MCTS simulations per move.
• Figure 4: Location of crucial moves by generation.Olivaw attributes high relevance to the conquest of the corners in the early generations, similarly to human beginners. In later generations, it shifts its "attention" towards the center of the board, as we would expect from a more experienced player.
• Figure 5: Olivaw vs Edax.a. Outcome of four $100$-game series between O-$1000$ and E$4$, E$6$, E$8$, and E$10$. b. Number of game-tree positions searched by E$10$ and O-$1000$ for each move, averaged over 10 games.