HERS: Hidden-Pattern Expert Learning for Risk-Specific Vehicle Damage Adaptation in Diffusion Models

TL;DR

HERS tackles the risk of semantically misleading, high-fidelity diffusion-generated vehicle damage in auto-insurance workflows by automatically learning risk-specific damage patterns through self-supervised domain experts. It combines domain-guided prompt synthesis, synthetic image generation, and LoRA-based damage experts that are merged into a unified multi-damage diffusion model, achieving improved text–image alignment and human-preferred realism across backbones. The method reportedly yields +5.5 percentage points in text fidelity and +2.3 percentage points in human preference over strong baselines, while enabling domain-aware, controllable generation without manual labels. The work underscores both opportunities for safer data augmentation in insurance workflows and risks of misuse, calling for robust auditing, watermarking, and cross-modal verification as essential safeguards for trustworthy deployment in high-stakes settings.

Abstract

Recent advances in text-to-image (T2I) diffusion models have enabled increasingly realistic synthesis of vehicle damage, raising concerns about their reliability in automated insurance workflows. The ability to generate crash-like imagery challenges the boundary between authentic and synthetic data, introducing new risks of misuse in fraud or claim manipulation. To address these issues, we propose HERS (Hidden-Pattern Expert Learning for Risk-Specific Damage Adaptation), a framework designed to improve fidelity, controllability, and domain alignment of diffusion-generated damage images. HERS fine-tunes a base diffusion model via domain-specific expert adaptation without requiring manual annotation. Using self-supervised image-text pairs automatically generated by a large language model and T2I pipeline, HERS models each damage category, such as dents, scratches, broken lights, or cracked paint, as a separate expert. These experts are later integrated into a unified multi-damage model that balances specialization with generalization. We evaluate HERS across four diffusion backbones and observe consistent improvements: plus 5.5 percent in text faithfulness and plus 2.3 percent in human preference ratings compared to baselines. Beyond image fidelity, we discuss implications for fraud detection, auditability, and safe deployment of generative models in high-stakes domains. Our findings highlight both the opportunities and risks of domain-specific diffusion, underscoring the importance of trustworthy generation in safety-critical applications such as auto insurance.

Figures (14)

• Figure 1: Qualitative comparison of HERS against existing diffusion-based baselines. Observe that HERS generates damage regions with higher visual fidelity and localized consistency. Fine-grained artifacts such as dents, cracks, and abrasions are better preserved—zoom in for enhanced visibility of subtle and complex damage patterns.
• Figure 2: Overview of the HERS Framework. HERS (Hidden-Pattern Expert Learning for Risk-Specific Damage Adaptation) auto-generates diverse, damage-specific image-text pairs using an LLM and a base T2I model—without requiring manual annotation. These pairs span typical vehicle parts, descriptive scene narratives, and physically implausible scenarios (examples shown in figure). Each damage type is modeled as a distinct damage, with corresponding LoRA experts trained and merged into a unified multi-damage diffusion model.
• Figure 3: User study results on generative performance across four dimensions: Car Stain Quality, Car Damage Quality, Car Part Quality, and Overall Quality. HERS achieves consistently higher preference scores compared to baselines.
• Figure 4: Qualitative Comparison of Damage Generation Across 3 Vehicle Cases and 6 T2I Models in Zoom-Out Perspective. Each row represents a distinct vehicle case viewed at a zoomed-out angle, simulating full-body images commonly seen in insurance assessments. The columns correspond to the outputs of six different T2I models: our proposed HERS (left-most), followed by VQ-Diffusion gu2022vector, Versatile Diffusion xu2022versatile, SDXL podell2023sdxl, MoLE zhu2024mole, and SELMA li2024selma. Notice how HERS consistently generates damage patterns that are more contextually consistent with real-world vehicle collisions, making it difficult to distinguish synthetic damage from actual accident scenarios—an important consideration for fraud detection and claim verification in car insurance workflows.
• Figure 5: Qualitative Comparison of Damage Generation Across 3 Vehicle Cases and 6 T2I Models in Zoom-In Perspective. Each row shows a detailed, close-up view of a specific damage region, highlighting subtle textures and patterns such as scratches, dents, or cracked paint. The columns correspond to outputs from six different T2I models: our proposed HERS (left-most), followed by VQ-Diffusion gu2022vector, Versatile Diffusion xu2022versatile, SDXL podell2023sdxl, MoLE zhu2024mole, and SELMA li2024selma. Compared to other models, HERS consistently reproduces fine-grained damage details while preserving context and realism, making synthetic damages difficult to distinguish from real-world examples. Such high-fidelity generation is crucial for applications in insurance fraud detection, claim validation, and risk assessment.