Comparing Reconstruction Attacks on Pretrained Versus Full Fine-tuned Large Language Model Embeddings on Homo Sapiens Splice Sites Genomic Data
Reem Al-Saidi, Erman Ayday, Ziad Kobti
TL;DR
This work investigates reconstruction attacks on embeddings from large language models applied to genomic data, focusing on how domain adaptation via fine-tuning affects privacy leakage. By applying a reconstruction pipeline to both pretrained and fine-tuned embeddings on the HS3D splice-site dataset and introducing DNA-specific tokenization plus extended position- and nucleotide-level metrics, the study provides a rigorous, apples-to-apples privacy comparison across architectures. It finds architecture-dependent privacy impacts: XLNet (+$0.198$), GPT-2 (+$0.098$), and BERT (+$0.078$) show improved privacy after fine-tuning, while RoBERTa (−$0.068$) and ERNIE (−$0.029$) experience degraded privacy, and ALBERT exhibits a zero-sum shift. These results highlight that domain adaptation can, in some cases, enhance privacy for genomic embeddings, while in others it redistributes leakage, underscoring the need for targeted protection strategies and further study of privacy-utility trade-offs with efficient fine-tuning methods.
Abstract
This study investigates embedding reconstruction attacks in large language models (LLMs) applied to genomic sequences, with a specific focus on how fine-tuning affects vulnerability to these attacks. Building upon Pan et al.'s seminal work demonstrating that embeddings from pretrained language models can leak sensitive information, we conduct a comprehensive analysis using the HS3D genomic dataset to determine whether task-specific optimization strengthens or weakens privacy protections. Our research extends Pan et al.'s work in three significant dimensions. First, we apply their reconstruction attack pipeline to pretrained and fine-tuned model embeddings, addressing a critical gap in their methodology that did not specify embedding types. Second, we implement specialized tokenization mechanisms tailored specifically for DNA sequences, enhancing the model's ability to process genomic data, as these models are pretrained on natural language and not DNA. Third, we perform a detailed comparative analysis examining position-specific, nucleotide-type, and privacy changes between pretrained and fine-tuned embeddings. We assess embeddings vulnerabilities across different types and dimensions, providing deeper insights into how task adaptation shifts privacy risks throughout genomic sequences. Our findings show a clear distinction in reconstruction vulnerability between pretrained and fine-tuned embeddings. Notably, fine-tuning strengthens resistance to reconstruction attacks in multiple architectures -- XLNet (+19.8\%), GPT-2 (+9.8\%), and BERT (+7.8\%) -- pointing to task-specific optimization as a potential privacy enhancement mechanism. These results highlight the need for advanced protective mechanisms for language models processing sensitive genomic data, while highlighting fine-tuning as a potential privacy-enhancing technique worth further exploration.