Scaling Vision Language Models for Pharmaceutical Long Form Video Reasoning on Industrial GenAI Platform

TL;DR

This work tackles the challenge of scaling Vision-Language Models for long-form pharmaceutical videos under realistic GPU, latency, and regulatory constraints. It introduces an industrial GenAI platform that ingests large-scale multimodal data and benchmarks over 40 VLMs on both standard and proprietary pharma datasets, revealing practical limits and deployment trade-offs. Four key findings emerge: multimodality generally improves performance, attention mechanism choices are hardware-dependent, long-form timing and keyframe alignment remain challenging for open and closed models, and splitting long videos can hinder rather than help efficiency. The study provides actionable guidance for researchers and practitioners to design and deploy scalable multimodal systems for long-form video understanding in regulated domains.

Abstract

Vision Language Models (VLMs) have shown strong performance on multimodal reasoning tasks, yet most evaluations focus on short videos and assume unconstrained computational resources. In industrial settings such as pharmaceutical content understanding, practitioners must process long-form videos under strict GPU, latency, and cost constraints, where many existing approaches fail to scale. In this work, we present an industrial GenAI framework that processes over 200,000 PDFs, 25,326 videos across eight formats (e.g., MP4, M4V, etc.), and 888 multilingual audio files in more than 20 languages. Our study makes three contributions: (i) an industrial large-scale architecture for multimodal reasoning in pharmaceutical domains; (ii) empirical analysis of over 40 VLMs on two leading benchmarks (Video-MME and MMBench) and proprietary dataset of 25,326 videos across 14 disease areas; and (iii) four findings relevant to long-form video reasoning: the role of multimodality, attention mechanism trade-offs, temporal reasoning limits, and challenges of video splitting under GPU constraints. Results show 3-8 times efficiency gains with SDPA attention on commodity GPUs, multimodality improving up to 8/12 task domains (especially length-dependent tasks), and clear bottlenecks in temporal alignment and keyframe detection across open- and closed-source VLMs. Rather than proposing a new "A+B" model, this paper characterizes practical limits, trade-offs, and failure patterns of current VLMs under realistic deployment constraints, and provide actionable guidance for both researchers and practitioners designing scalable multimodal systems for long-form video understanding in industrial domains.

Figures (7)

• Figure 1: System architecture of our GenAI platform for Natural Language (NL) search integrating LLMs, ALMs, and VLMs. The platform processed 25,326 videos, 888 audios covering > 20 languages.
• Figure 2: Multimodality matters. Combining metadata and voiceovers using LLM/ALMs could improve VLM summaries and understanding. Especially effective for longer videos, and for tasks like action recognition, object reasoning, and OCR, but may negatively impact temporal reasoning tasks.
• Figure 3: Time Localization Challenges for Open-Source and Closed-Source VLMs. Both top open-source and commercial models struggle with key frame detection, showing low accuracy (5-35%) and incorrect timestamps. Summaries are much more accurate, ranging from 75-95%.
• Figure 4: Comparison of the Video-MME fu2024video and MMBench-Video datasets MMBench in terms of video categories and duration distributions. The Video-MME dataset consists of 900 videos spanning six primary visual domains with 30 subfields, categorized into 300 short-term (<2 min), 300 medium-term (4-15 min), and 300 long-term (30-60 min) videos. In contrast, the MMBench-Video dataset comprises approximately 609 videos across 16 major categories, with durations ranging all from 30 seconds to 6 minutes.
• Figure 5: Knowledge graph for summary and key frames using Qwen and Gemini models. The knowledge graph visualizes the comparison between Gemini-2-Flash and Qwen-7B in summarizing a 'Snow White' stage performance. Each key frame from Gemini-2 Flash is marked in light blue and video summary in dark blue, while Qwen-7B's key frames are in light red, and video summary in red. The central node represents the key frames, with connections showing their relationships to each model's summary. Gemini-2-Flash emphasizes narrative elements such as the Magic Mirror, the Evil Queen, and the climax involving Snow White's revival, while Qwen-7B structures the story around broad thematic transitions like character introductions, forest scenes, and musical elements. This graph presents a structured comparison of the keyframes extracted by Gemini-2-Flash and Qwen-7B from a 'Snow White' performance. The blue nodes represent Gemini-2-Flash’s emphasis on theatrical storytelling, focusing on individual character moments, while the red nodes highlight Qwen-7B’s broader narrative structure, including interactions between Snow White and supporting characters. Additionally, the red nodes are more widely distributed, whereas the blue nodes are clustered more closely, indicating a difference in granularity and focus.