Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

Ming Cheng, Jiaying Gong, Hoda Eldardiry

TL;DR

Sci-LoRA tackles cross-domain lay paraphrasing by learning a mixture of domain-specific LoRA adapters and dynamically weighting them at inference without domain labels. The approach combines domain-aware adapters trained per domain, a contrastive text encoder to produce discriminative domain representations, and a dynamic fusion mechanism that balances specialized and general knowledge. Across twelve domains and five public datasets, Sci-LoRA consistently outperforms strong baselines on a wide set of metrics and receives favorable human evaluations for comprehensiveness and fluency, while ablations confirm the necessity of each component. This work enables more scalable, cross-domain access to scientific content for lay audiences and supports interdisciplinary communication without explicit domain annotations at inference time.

Abstract

Lay paraphrasing aims to make scientific information accessible to audiences without technical backgrounds. However, most existing studies focus on a single domain, such as biomedicine. With the rise of interdisciplinary research, it is increasingly necessary to comprehend knowledge spanning multiple technical fields. To address this, we propose Sci-LoRA, a model that leverages a mixture of LoRAs fine-tuned on multiple scientific domains. In particular, Sci-LoRA dynamically generates and applies weights for each LoRA, enabling it to adjust the impact of different domains based on the input text, without requiring explicit domain labels. To balance domain-specific knowledge and generalization across various domains, Sci-LoRA integrates information at both the data and model levels. This dynamic fusion enhances the adaptability and performance across various domains. Experimental results across twelve domains on five public datasets show that Sci-LoRA significantly outperforms state-of-the-art large language models and demonstrates flexible generalization and adaptability in cross-domain lay paraphrasing.

Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

TL;DR

Sci-LoRA tackles cross-domain lay paraphrasing by learning a mixture of domain-specific LoRA adapters and dynamically weighting them at inference without domain labels. The approach combines domain-aware adapters trained per domain, a contrastive text encoder to produce discriminative domain representations, and a dynamic fusion mechanism that balances specialized and general knowledge. Across twelve domains and five public datasets, Sci-LoRA consistently outperforms strong baselines on a wide set of metrics and receives favorable human evaluations for comprehensiveness and fluency, while ablations confirm the necessity of each component. This work enables more scalable, cross-domain access to scientific content for lay audiences and supports interdisciplinary communication without explicit domain annotations at inference time.

Abstract

Lay paraphrasing aims to make scientific information accessible to audiences without technical backgrounds. However, most existing studies focus on a single domain, such as biomedicine. With the rise of interdisciplinary research, it is increasingly necessary to comprehend knowledge spanning multiple technical fields. To address this, we propose Sci-LoRA, a model that leverages a mixture of LoRAs fine-tuned on multiple scientific domains. In particular, Sci-LoRA dynamically generates and applies weights for each LoRA, enabling it to adjust the impact of different domains based on the input text, without requiring explicit domain labels. To balance domain-specific knowledge and generalization across various domains, Sci-LoRA integrates information at both the data and model levels. This dynamic fusion enhances the adaptability and performance across various domains. Experimental results across twelve domains on five public datasets show that Sci-LoRA significantly outperforms state-of-the-art large language models and demonstrates flexible generalization and adaptability in cross-domain lay paraphrasing.

Paper Structure

This paper contains 28 sections, 9 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Lay Paraphrasing
  4. Mixture of Loras
  5. Methodology
  6. Background
  7. Low-Rank Adaptation
  8. Problem Formulation
  9. Sci-LoRA Framework
  10. Domain LoRAs Training
  11. Adapter Weight Generator
  12. Text Encoder
  13. Weight Generator
  14. Dynamic LoRA Fusion
  15. Experiments
  16. ...and 13 more sections

Figures (3)

  • Figure 1: The Sci-LoRA framework including the adapter weight generator and dynamic LoRA fusion. All text encoder modules are the same text encoder trained on a subset of training data across all domains.
  • Figure 2: t-SNE visualization (before contrastive learning and after contrastive learning) of text embeddings from CELLS, eLife, SciTechNews, PLOS and ENG in VTechAGP datasets.
  • Figure 3: Human evaluation guideline.