ctELM: Decoding and Manipulating Embeddings of Clinical Trials with Embedding Language Models

TL;DR

This work introduces ctELM, an open-source Embedding Language Model framework that aligns a base LLM to clinical-trial embeddings, enabling reconstruction and generation of abstracts from embeddings. It defines five domain-focused tasks and uses LoRA-based fine-tuning of a lightweight adapter to achieve accurate embedding-to-text mappings, validated through semantic-consistency metrics, interpolations, and human plus LLM-based plausibility tests. ctELM demonstrates robustness across data scales, base and embedding models, and remains responsive to Concept Activation Vectors representing sex and age, enabling controlled generation. The study provides extensive ablations, a public implementation, and a framework aimed at broader applicability of embedding-space alignment in biomedicine and beyond, with considerations for ethics and generalization.

Abstract

Text embeddings have become an essential part of a variety of language applications. However, methods for interpreting, exploring and reversing embedding spaces are limited, reducing transparency and precluding potentially valuable generative use cases. In this work, we align Large Language Models to embeddings of clinical trials using the recently reported Embedding Language Model (ELM) method. We develop an open-source, domain-agnostic ELM architecture and training framework, design training tasks for clinical trials, and introduce an expert-validated synthetic dataset. We then train a series of ELMs exploring the impact of tasks and training regimes. Our final model, ctELM, can accurately describe and compare unseen clinical trials from embeddings alone and produce plausible clinical trials from novel vectors. We further show that generated trial abstracts are responsive to moving embeddings along concept vectors for age and sex of study subjects. Our public ELM implementation and experimental results will aid the alignment of Large Language Models to embedding spaces in the biomedical domain and beyond.

Figures (24)

• Figure 1: The data generation and training pipeline for ctELM.
• Figure 2: Moving embeddings along a Concept Activation Vector for sex of trial subjects changes the observed sex in abstracts generated by ctELM. The value $\alpha$ is the coefficient of the added sex vector and thus represents concept strength. Trial subject sex (y-axis, left) refers to the number of trials identified as each sex among a group of 50. REF is sex extracted from original abstracts. Semantic Consistency is shown in black lines (y-axis, right).
• Figure 3: Moving embeddings along Concept Activation Vectors for age of trial subjects changes the observed age in abstracts generated by ctELM. The value $\alpha$ is the coefficient of the added age vector. Trial subject age (y-axis, left) refers to the identified age of each trial (each depicted as a point). Box and whisker plots show minima, maxima, medians, and inter-quartile ranges of identified age. Note that horizontal jitter is employed for each discrete $\alpha$ value; the x position of each point within its strip is thus not meaningful. REF is the original abstracts with age extracted directly. Semantic Consistency is shown in black lines (y-axis, right).
• Figure 4: The prompt template for generating plain language summary.
• Figure 5: The scatter plot between two physicians' annotated scores with Pearson correlation coefficient ($r$) results.