MeetBench-XL: Calibrated Multi-Dimensional Evaluation and Learned Dual-Policy Agents for Real-Time Meetings

TL;DR

MeetAll provides an enterprise-grounded bilingual multimodal meeting dataset and a calibrated MeetBench-XL evaluation framework to align AI meeting assistants with real-world workflows. It introduces MeetMaster-XL, a learned dual-policy agent that jointly optimizes routing and tool invocation, achieving deployment-ready performance in on-premise settings and live field pilots. The dataset employs an enterprise-informed, multi-axes annotation scheme to cover cognitive load, temporal context, domain knowledge, and actionable task execution, with ground-truth annotations refined by humans. Field deployments demonstrate improved latency, quality, and cost-efficiency relative to baselines, underscoring practical impact for production-grade, privacy-conscious meeting assistants.

Abstract

Enterprise meeting environments require AI assistants that handle diverse operational tasks, from rapid fact checking during live discussions to cross meeting analysis for strategic planning, under strict latency, cost, and privacy constraints. Existing meeting benchmarks mainly focus on simplified question answering and fail to reflect real world enterprise workflows, where queries arise organically from multi stakeholder collaboration, span long temporal contexts, and require tool augmented reasoning. We address this gap through a grounded dataset and a learned agent framework. First, we introduce MeetAll, a bilingual and multimodal corpus derived from 231 enterprise meetings totaling 140 hours. Questions are injected using an enterprise informed protocol validated by domain expert review and human discriminability studies. Unlike purely synthetic benchmarks, this protocol is grounded in four enterprise critical dimensions: cognitive load, temporal context span, domain expertise, and actionable task execution, calibrated through interviews with stakeholders across finance, healthcare, and technology sectors. Second, we propose MeetBench XL, a multi dimensional evaluation protocol aligned with human judgment that measures factual fidelity, intent alignment, response efficiency, structural clarity, and completeness. Third, we present MeetMaster XL, a learned dual policy agent that jointly optimizes query routing between fast and slow reasoning paths and tool invocation, including retrieval, cross meeting aggregation, and web search. A lightweight classifier enables accurate routing with minimal overhead, achieving a superior quality latency tradeoff over single model baselines. Experiments against commercial systems show consistent gains, supported by ablations, robustness tests, and a real world deployment case study.Resources: https://github.com/huyuelin/MeetBench.

Figures (3)

• Figure 1: Overview of MeetAll dataset and MeetBench-XL benchmark grounded in enterprise meeting workflows. MeetAll contains 231 meetings (140 hours) with enterprise-validated question injection spanning operational fact-checking, cross-meeting synthesis, and actionable planning tasks. MeetBench-XL evaluates assistants on five enterprise-prioritized dimensions calibrated to human judgment. MeetMaster-XL achieves deployment-ready performance through learned policy optimization.
• Figure 2: Quality evaluation metric for MeetBench-XL. The adapted evaluator aggregates five equally weighted dimensions (Factual, User Need, Conciseness, Structure, Completeness) to produce an overall score; calibration details are in §\ref{['sec:calibration']}.
• Figure 3: MeetMaster-XL employs a dual-process architecture with a learned router: queries are processed in parallel by a fast Talker and a reasoning Planner. After ASRmachacek2023turningradford2023whisperwang2023wekws, the system predicts type and complexity and routes to the appropriate pathway.