Performance of the SafeTerm AI-Based MedDRA Query System Against Standardised MedDRA Queries

TL;DR

The paper tackles the challenge of efficiently generating relevant MedDRA terms for Standardised MedDRA Queries (SMQs) in pharmacovigilance. It introduces SafeTerm Automated MedDRA Query (AMQ), a fully unsupervised, embedding-based pipeline that retrieves and ranks MedDRA PTs using a data-driven similarity threshold. Validation on MedDRA v28.1 Tier-1 SMQs shows strong recall at moderate thresholds and improved precision with higher thresholds, with the auto Knee-based threshold balancing recall and precision. The approach demonstrates robust, scalable performance across SMQs and supports practical, version-agnostic automated query generation with human-in-the-loop refinement.

Abstract

In pre-market drug safety review, grouping related adverse event terms into SMQs or OCMQs is critical for signal detection. We assess the performance of SafeTerm Automated Medical Query (AMQ) on MedDRA SMQs. The AMQ is a novel quantitative artificial intelligence system that understands and processes medical terminology and automatically retrieves relevant MedDRA Preferred Terms (PTs) for a given input query, ranking them by a relevance score (0-1) using multi-criteria statistical methods. The system (SafeTerm) embeds medical query terms and MedDRA PTs in a multidimensional vector space, then applies cosine similarity, and extreme-value clustering to generate a ranked list of PTs. Validation was conducted against tier-1 SMQs (110 queries, v28.1). Precision, recall and F1 were computed at multiple similarity-thresholds, defined either manually or using an automated method. High recall (94%)) is achieved at moderate similarity thresholds, indicative of good retrieval sensitivity. Higher thresholds filter out more terms, resulting in improved precision (up to 89%). The optimal threshold (0.70)) yielded an overall recall of (48%) and precision of (45%) across all 110 queries. Restricting to narrow-term PTs achieved slightly better performance at an increased (+0.05) similarity threshold, confirming increased relatedness of narrow versus broad terms. The automatic threshold (0.66) selection prioritizes recall (0.58) to precision (0.29). SafeTerm AMQ achieves comparable, satisfactory performance on SMQs and sanitized OCMQs. It is therefore a viable supplementary method for automated MedDRA query generation, balancing recall and precision. We recommend using suitable MedDRA PT terminology in query formulation and applying the automated threshold method to optimise recall. Increasing similarity scores allows refined, narrow terms selection.

Figures (3)

• Figure 1: Mean $\pm$ SD: Precision, Recall, F1 versus similarity cut-off (across 110 Tier-1 SMQs).
• Figure 2: Maximum F1 score versus SMQ gold set size for Narrow + Broad and Narrow-only Term Lists.
• Figure 3: Mean $\pm$ SD: Precision, Recall, F1 versus similarity cut-off for Narrow Terms Retrieval (across SMQs).