Optimizing Multimodal LLMs for Egocentric Video Understanding: A Solution for the HD-EPIC VQA Challenge
Sicheng Yang, Yukai Huang, Shitong Sun, Weitong Cai, Jiankang Deng, Jifei Song, Zhensong Zhang
TL;DR
The paper tackles the problem of egocentric video QA on the HD-EPIC benchmark, which demands long-range temporal reasoning and fine-grained action understanding. It presents a holistic framework that combines query/choice pre-processing, domain-specific fine-tuning of Qwen2.5-VL-7B-Instruct, a Temporal Chain-of-Thought prompting strategy, and robust post-processing with ensembling. Key contributions include structured input/output processing, temporal reasoning guidance, and an ensembling-plus-cleaning pipeline that achieves 41.6% accuracy on HD-EPIC VQA and shows a +3.0% improvement from temporal reasoning. The work demonstrates that holistic pipeline optimization across data, model adaptation, and guided reasoning is crucial for demanding egocentric video understanding, offering a strong baseline and open-source resources for future research and practical applications.
Abstract
Multimodal Large Language Models (MLLMs) struggle with complex video QA benchmarks like HD-EPIC VQA due to ambiguous queries/options, poor long-range temporal reasoning, and non-standardized outputs. We propose a framework integrating query/choice pre-processing, domain-specific Qwen2.5-VL fine-tuning, a novel Temporal Chain-of-Thought (T-CoT) prompting for multi-step reasoning, and robust post-processing. This system achieves 41.6% accuracy on HD-EPIC VQA, highlighting the need for holistic pipeline optimization in demanding video understanding. Our code, fine-tuned models are available at https://github.com/YoungSeng/Egocentric-Co-Pilot.