From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)

TL;DR

This work addresses the need for efficient, auditable extraction of clinically meaningful highlights from long pharmaceutical videos by introducing an industrially deployable Infinite Video-to-Video Clips Generation framework. It combines Audio Language Models and Vision Language Models with a novel Cut & Merge post-processing pipeline and role-based prompt injections to produce compliant, high-quality extractive clips at scale. Evaluations on the Video-MME benchmark and a large proprietary pharma dataset show substantial gains in speed and cost (3–4× faster, ~4× cheaper) while improving clip coherence and informativeness. The approach enhances content reuse and regulatory compliance in life sciences, enabling faster review and dissemination of medical and educational video assets.

Abstract

Vision Language Models (VLMs) are poised to revolutionize the digital transformation of pharmacyceutical industry by enabling intelligent, scalable, and automated multi-modality content processing. Traditional manual annotation of heterogeneous data modalities (text, images, video, audio, and web links), is prone to inconsistencies, quality degradation, and inefficiencies in content utilization. The sheer volume of long video and audio data further exacerbates these challenges, (e.g. long clinical trial interviews and educational seminars). Here, we introduce a domain adapted Video to Video Clip Generation framework that integrates Audio Language Models (ALMs) and Vision Language Models (VLMs) to produce highlight clips. Our contributions are threefold: (i) a reproducible Cut & Merge algorithm with fade in/out and timestamp normalization, ensuring smooth transitions and audio/visual alignment; (ii) a personalization mechanism based on role definition and prompt injection for tailored outputs (marketing, training, regulatory); (iii) a cost efficient e2e pipeline strategy balancing ALM/VLM enhanced processing. Evaluations on Video MME benchmark (900) and our proprietary dataset of 16,159 pharmacy videos across 14 disease areas demonstrate 3 to 4 times speedup, 4 times cost reduction, and competitive clip quality. Beyond efficiency gains, we also report our methods improved clip coherence scores (0.348) and informativeness scores (0.721) over state of the art VLM baselines (e.g., Gemini 2.5 Pro), highlighting the potential of transparent, custom extractive, and compliance supporting video summarization for life sciences.

Figures (11)

• Figure 1: Solution architecture blueprint of the underlying LLM/VLM tech stack for video clip generation.
• Figure 2: Qualitative comparison of our Infinite Video-to-Video Clips pipeline against frame-based approaches (e.g. Runway Gen-2 runwayml2023gen2). Our method supports arbitrary input durations, allows user-defined output lengths, automatically extracts agenda-relevant segments, adds subtitles and vertical playback, while overcoming choppy transitions and frame skipping/freezing, e.g, (e).
• Figure 3: Processing Time Comparison: Gemini 2.5 Pro / Flash vs Our Methods for generating video clips script on VideoMME Long Video Dataset. Gemini Pro is the slowest (avg. $\sim$ 120s/video), with peaks on longer videos (e.g.,380s). Flash is faster ($\sim$80–85s) but still slower than our method ($\sim$30–55s), except for one outlier (413s). Our method is in general 3-4x faster.
• Figure 4: Comparison between the Gemini 2.5 Flash, Pro vs our, based on the number of select segments and the quality. Here, we assess quality based on factors like segment length / numbers or the presence of coherent text. Flash often returns many but fragment segments (e.g. “Video ID: tslKtm6Le1s”: 85 piece of segments). Pro tends to pick fewer, longer segments (reflected in its lower segment counts but higher average durations). Our method achieves balanced selection 4.37 segments vs. 7.38 (Gemini 2.5 Pro) and 13.30 (Gemini 2.5 Flash) for final clips.
• Figure 5: Qualitative comparison of Whisper V2 vs. V3: transcription accuracy on LibriSpeech panayotov2015librispeech test-clean and inference speed on AWS hardware. Whisper V3’s performance is more sensitive to GPU type—achieving roughly 4–8× speedups and higher Accuracy compared to V2, but cuts speech more aggressively, resulting in increased sentence fragmentation that complicates downstream timestamp alignment and segment merging.