Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Augmentation for High-Fidelity Generation of CAR-T/NK Immunological Synapse Images

Xiang Zhang, Boxuan Zhang, Alireza Naghizadeh, Mohab Mohamed, Dongfang Liu, Ruixiang Tang, Dimitris Metaxas, Dongfang Liu

TL;DR

The paper tackles the data bottleneck in CAR-T/NK immunological synapse imaging by introducing two complementary augmentation pipelines. IAAA provides instance-aware, policy-driven augmentation on real images with precise segmentation masks, while SAAA extensibly generates new masks and images via a diffusion-based mask generator and a Pix2Pix semantic synthesizer, enabling scalable data production. Across SCOs, MCOs, and final IS images, the methods yield competitive $FID$/$KID$ scores and consistently improve bounding-box detection and instance segmentation metrics ($AP_{50}$, $AP_{75}$, $AP_s$) compared to GAN baselines. Collectively, these approaches enhance robustness and accuracy of IS quantification, supporting the development of imaging-based biomarkers for predicting patient responses to CAR-T/NK therapies and enabling scalable datasets for biomedical vision models.

Abstract

Chimeric antigen receptor (CAR)-T and NK cell immunotherapies have transformed cancer treatment, and recent studies suggest that the quality of the CAR-T/NK cell immunological synapse (IS) may serve as a functional biomarker for predicting therapeutic efficacy. Accurate detection and segmentation of CAR-T/NK IS structures using artificial neural networks (ANNs) can greatly increase the speed and reliability of IS quantification. However, a persistent challenge is the limited size of annotated microscopy datasets, which restricts the ability of ANNs to generalize. To address this challenge, we integrate two complementary data-augmentation frameworks. First, we employ Instance Aware Automatic Augmentation (IAAA), an automated, instance-preserving augmentation method that generates synthetic CAR-T/NK IS images and corresponding segmentation masks by applying optimized augmentation policies to original IS data. IAAA supports multiple imaging modalities (e.g., fluorescence and brightfield) and can be applied directly to CAR-T/NK IS images derived from patient samples. In parallel, we introduce a Semantic-Aware AI Augmentation (SAAA) pipeline that combines a diffusion-based mask generator with a Pix2Pix conditional image synthesizer. This second method enables the creation of diverse, anatomically realistic segmentation masks and produces high-fidelity CAR-T/NK IS images aligned with those masks, further expanding the training corpus beyond what IAAA alone can provide. Together, these augmentation strategies generate synthetic images whose visual and structural properties closely match real IS data, significantly improving CAR-T/NK IS detection and segmentation performance. By enhancing the robustness and accuracy of IS quantification, this work supports the development of more reliable imaging-based biomarkers for predicting patient response to CAR-T/NK immunotherapy.

Data Augmentation for High-Fidelity Generation of CAR-T/NK Immunological Synapse Images

TL;DR

The paper tackles the data bottleneck in CAR-T/NK immunological synapse imaging by introducing two complementary augmentation pipelines. IAAA provides instance-aware, policy-driven augmentation on real images with precise segmentation masks, while SAAA extensibly generates new masks and images via a diffusion-based mask generator and a Pix2Pix semantic synthesizer, enabling scalable data production. Across SCOs, MCOs, and final IS images, the methods yield competitive / scores and consistently improve bounding-box detection and instance segmentation metrics (, , ) compared to GAN baselines. Collectively, these approaches enhance robustness and accuracy of IS quantification, supporting the development of imaging-based biomarkers for predicting patient responses to CAR-T/NK therapies and enabling scalable datasets for biomedical vision models.

Abstract

Chimeric antigen receptor (CAR)-T and NK cell immunotherapies have transformed cancer treatment, and recent studies suggest that the quality of the CAR-T/NK cell immunological synapse (IS) may serve as a functional biomarker for predicting therapeutic efficacy. Accurate detection and segmentation of CAR-T/NK IS structures using artificial neural networks (ANNs) can greatly increase the speed and reliability of IS quantification. However, a persistent challenge is the limited size of annotated microscopy datasets, which restricts the ability of ANNs to generalize. To address this challenge, we integrate two complementary data-augmentation frameworks. First, we employ Instance Aware Automatic Augmentation (IAAA), an automated, instance-preserving augmentation method that generates synthetic CAR-T/NK IS images and corresponding segmentation masks by applying optimized augmentation policies to original IS data. IAAA supports multiple imaging modalities (e.g., fluorescence and brightfield) and can be applied directly to CAR-T/NK IS images derived from patient samples. In parallel, we introduce a Semantic-Aware AI Augmentation (SAAA) pipeline that combines a diffusion-based mask generator with a Pix2Pix conditional image synthesizer. This second method enables the creation of diverse, anatomically realistic segmentation masks and produces high-fidelity CAR-T/NK IS images aligned with those masks, further expanding the training corpus beyond what IAAA alone can provide. Together, these augmentation strategies generate synthetic images whose visual and structural properties closely match real IS data, significantly improving CAR-T/NK IS detection and segmentation performance. By enhancing the robustness and accuracy of IS quantification, this work supports the development of more reliable imaging-based biomarkers for predicting patient response to CAR-T/NK immunotherapy.
Paper Structure (32 sections, 1 equation, 26 figures, 5 tables)

This paper contains 32 sections, 1 equation, 26 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Greedy AutoAugment
  4. Search Space
  5. Score
  6. Search Algorithm
  7. Wasserstein AutoEncoder
  8. Artificial Background Images
  9. Placement of Cells on the Background
  10. Structure of Instance Aware Automatic Augmentation (IAAA)
  11. Results
  12. Evaluation Metrics
  13. Data Preparation for Original Datasets
  14. Data Preparation for Mixed Datasets
  15. Evaluation of the Single-Cell Object (SCO) Image Quality Generation by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID)
  16. ...and 17 more sections

Figures (26)

  • Figure 1: The removal of cells from a real image to create an artificially generated background and the placement of cells on the artificially generated background. (a) shows a sample of the real dataset from the Neural dataset. (b) shows the process of removing the first cell from the image. (c) (d) (e) (f) shows the process of removing four other cells. (g) shows an empty background. (h) shows the process of adding the first cell to the background. (i) (j) (k) and (l) show the process of adding four other cells.
  • Figure 2: Experimental diagram. (a) shows the overall structure of the IAAA technique. (b) Pipeline for creating background images. (c) Pipeline for adding cells on backgrounds. Gaussian filtering removes and adds patches seamlessly. (d) Pipeline for image generation.
  • Figure 3: Representative image generation using SCOs. The (a) and (b) represent the artificial images and their masks generated by our method. The (c), and (d) represent reference (real) images which are accompanied by their associated masks. We zoom into different regions of the images for better visibility of SCOs. The FID and KID scores of SCOs, are presented in (e), and (f) using the CAR-T/NK dataset. For comparison, the quality of images from IAAA is measured against four GAN methods (DCGAN, BigGAN-SD, BigGAN-Diff, and StyleGAN2-Diff).
  • Figure 4: Representative image generation using MCOs. The (a) and (b) represent the artificial images and their masks generated by our method. The (c), and (d) represent reference (real) images which are accompanied by their associated masks. We zoom into different regions of the images for better visibility of MCOs. The FID and KID scores of MCOs are presented in (e), and (f) using the CAR-T/NK dataset. For comparison, the quality of images from IAAA is measured against four GAN methods (DCGAN, BigGAN-SD, BigGAN-Diff, and StyleGAN2-Diff).
  • Figure 5: Representative image generation by MCOs using the CAR-T/NK dataset. The (a) and (b) represent the artificial images and their masks generated by our method. The (c) represents reference (real) images which are accompanied by their associated masks. The FID and KID scores are presented in (d), and (e) using the CAR-T/NK dataset. For comparison, the quality of images from IAAA is measured against four GAN methods (DCGAN, BigGAN-SD, BigGAN-Diff, and StyleGAN2-Diff).
  • ...and 21 more figures