Formal Semantic Control over Language Models

TL;DR

This work investigates how to render language representations more semantically and geometrically interpretable by shaping latent space geometry within a Variational Autoencoder framework. It develops a formal semantic geometry that treats semantic features as convex cones in latent space and introduces a SRL-Conditional VAE to integrate predicate-argument structure with generation. It extends this by introducing a dual-encoder syntax–semantic VAE and discrete latent spaces via VQ-VAE with T5VQVAE, along with a quasi-symbolic NLI framework that encodes explanatory inference patterns (AMR/AST) as latent subspaces governed by Neural Tangent Kernel theory. Across sentence-level and reasoning-level tasks, the methods demonstrate improved disentanglement, controllability, and quasi-symbolic inference, enabling finer-grained manipulation of outputs while bridging distributional representations with formal semantics. The work offers a principled path toward interpretable and controllable language models with potential impact on safety, reliability, and transferability in NLP systems.

Abstract

This thesis advances semantic representation learning to render language representations or models more semantically and geometrically interpretable, and to enable localised, quasi-symbolic, compositional control through deliberate shaping of their latent space geometry. We pursue this goal within a VAE framework, exploring two complementary research directions: (i) Sentence-level learning and control: disentangling and manipulating specific semantic features in the latent space to guide sentence generation, with explanatory text serving as the testbed; and (ii) Reasoning-level learning and control: isolating and steering inference behaviours in the latent space to control NLI. In this direction, we focus on Explanatory NLI tasks, in which two premises (explanations) are provided to infer a conclusion. The overarching objective is to move toward language models whose internal semantic representations can be systematically interpreted, precisely structured, and reliably directed. We introduce a set of novel theoretical frameworks and practical methodologies, together with corresponding experiments, to demonstrate that our approaches enhance both the interpretability and controllability of latent spaces for natural language across the thesis.

Figures (66)

• Figure 1: This thesis focuses on the "semantic and geometrical" interpretability and "fine-grained, localised, quasi-symbolic, compositional" generative control of Transformer Language Models (TLMs) by shaping the latent space geometry within the Variational AutoEncoder (VAE) framework.
• Figure 2: Overview of research questions and their solutions.
• Figure 3: Overview of the thesis structure, including dependencies between research questions and chapters. Chapters 2 and 3 relate directly to all chapters by investigating the background and literature review.
• Figure 4: An example of an explanation-based entailment tree from EntailmentBank dalvi2021explaining. In this thesis, the focus is restricted to single-step explanatory inference, where two premises are combined to infer a conclusion.
• Figure 5: AMR argument substitution: the explanatory inference behaviour can be defined as subgraph substitution zhang2023type.