LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation

TL;DR

The paper addresses the challenge of integrating video LLM world knowledge into scalable video recommendation. LinkedOut extracts world-knowledge tokens from raw video frames using a cross-layer mixture-of-experts and a store-and-retrieve architecture to achieve fast, multi-video inference without fine-tuning. It introduces a Layer Token Compressor and Cross-Layer Knowledge MoE to fuse information across VLLM depths, balancing fine-grained visual cues with high-level semantics. Experiments on MicroLens datasets show state-of-the-art results and highlight the practical efficiency and interpretability of the approach.

Abstract

Video Large Language Models (VLLMs) unlock world-knowledge-aware video understanding through pretraining on internet-scale data and have already shown promise on tasks such as movie analysis and video question answering. However, deploying VLLMs for downstream tasks such as video recommendation remains challenging, since real systems require multi-video inputs, lightweight backbones, low-latency sequential inference, and rapid response. In practice, (1) decode-only generation yields high latency for sequential inference, (2) typical interfaces do not support multi-video inputs, and (3) constraining outputs to language discards fine-grained visual details that matter for downstream vision tasks. We argue that these limitations stem from the absence of a representation that preserves pixel-level detail while leveraging world knowledge. We present LinkedOut, a representation that extracts VLLM world knowledge directly from video to enable fast inference, supports multi-video histories, and removes the language bottleneck. LinkedOut extracts semantically grounded, knowledge-aware tokens from raw frames using VLLMs, guided by promptable queries and optional auxiliary modalities. We introduce a cross-layer knowledge fusion MoE that selects the appropriate level of abstraction from the rich VLLM features, enabling personalized, interpretable, and low-latency recommendation. To our knowledge, LinkedOut is the first VLLM-based video recommendation method that operates on raw frames without handcrafted labels, achieving state-of-the-art results on standard benchmarks. Interpretability studies and ablations confirm the benefits of layer diversity and layer-wise fusion, pointing to a practical path that fully leverages VLLM world-knowledge priors and visual reasoning for downstream vision tasks such as recommendation.

Figures (5)

• Figure 1: Conceptual differences of our LinkedOut and other methods. Existing approaches using video models either as a classifier or a feature extractor (i.e., (a) and (b)) which are trained on task specific data without LLM/world knowledge. Other solutions such as (c) VLLM for content summarization, (d) VLLM direct video ranking, or (e) VLLM for token level feature extraction, they can neither fully utilize the world knowledge from VLLM nor are computationally feasible for recommendation online serving. Our LinkedOut approach is able to fully utilize both pixel level visual information with VLLM knowledge. Videos are eventually transferred to feature vectors for fast online recommendation serving.
• Figure 2: LinkedOut framework. Raw video frames are tokenized and projected into a pretrained video LLM. The optional text queries (e.g. transcripts, metadata, and dynamic context) are co-tokenized and injected to guide extraction. As tokens traverse lower to higher transformer layers, cross-modal attention enriches the sequence with both existing visual/text tokens and LLM-generated knowledge tokens. A Layer-token Compressor Expert condenses token sequences, and a Layer Knowledge Fuser MoE gates across layers to adaptively select the appropriate semantic depth, yielding a unified, world-knowledge-aware video embedding. At serving time, user features and video embeddings are passed to a lightweight recommendation model to produce a relevance score. The promptable design supports flexible modality fusion and selective fine-tuning, enabling personalized, interpretable, and cold-start-robust video recommendation.
• Figure 3: Overview of the Cross-Layer Knowledge-Fusion MoE. Token Compressor Expert receives existing and generated token features. A token-merge module aggregates redundant signals at the feature level, followed by normalization and projection. A lightweight selection expert predicts scores to reweight tokens; the two streams are then merged to yield a compact, semantically faithful representation for layer k. Cross-Layer Knowledge MoE Fuser treats each layer as an expert. Layer encoders summarize the compressed tokens, and layer-level soft-gating produces context-conditioned mixture weights that adaptively combine information across layers. The resulting LinkedOut feature fuses fine-grained visual details with world knowledge, providing a single, retrieval-ready embedding that improves efficiency while preserving recommendation-relevant semantics.
• Figure 4: Probability density distribution of layer-wise MoE gate weights via kernel density estimation (KDE). Each subplot shows the distribution of normalized contribution values for a transformer layer (L0, L4, L8, L12, L16, L20). Distribution width and peak location reveal utilization patterns, where L8 shows the broadest distribution centered at 0.4, while L4 and L16 exhibit more concentrated distributions with relatively smaller contributions.
• Figure 5: Statistical summary of layer-wise MoE gate contributions. Box plots show distribution of normalized contribution values per layer, with red lines (median) and blue dashed lines (mean). Annotated percentages are average contributions, where L8 dominates at 40.9%, followed by L20 (20.5%) and L0 (16.4%), while L4 (0.5%) and L16 (6.7%) contribute relatively less.