TVWorld: Foundations for Remote-Control TV Agents

TL;DR

This work introduces TVWorld, a static, graph-based framework to study focus-based remote-control TV navigation, paired with TVWorld-N for topology-aware navigation and TVWorld-G for focus grounding. It reveals that current LVLMs struggle with topology-aware planning in TV RC tasks and proposes a two-stage Topology-Aware Training regime to inject focus and topology understanding, culminating in TVTheseus, a TV navigation foundation model. TVTheseus achieves strong out-of-domain performance on TVWorld-N ($68.3\%$ SR) and high grounding accuracy on TVWorld-G ($81.8\%$ Acc@0.5), outperforming closed-source baselines and showcasing robust topology-guided planning. The approach provides valuable benchmarks and training paradigms for developing deployable, topology-aware TV-use agents in remote-control settings, with implications for broader GUI control research beyond pointer-based interactions.

Abstract

Recent large vision-language models (LVLMs) have demonstrated strong potential for device control. However, existing research has primarily focused on point-and-click (PnC) interaction, while remote-control (RC) interaction commonly encountered in everyday TV usage remains largely underexplored. To fill this gap, we introduce \textbf{TVWorld}, an offline graph-based abstraction of real-world TV navigation that enables reproducible and deployment-free evaluation. On this basis, we derive two complementary benchmarks that comprehensively assess TV-use capabilities: \textbf{TVWorld-N} for topology-aware navigation and \textbf{TVWorld-G} for focus-aware grounding. These benchmarks expose a key limitation of existing agents: insufficient topology awareness for focus-based, long-horizon TV navigation. Motivated by this finding, we propose a \emph{Topology-Aware Training} framework that injects topology awareness into LVLMs. Using this framework, we develop \textbf{TVTheseus}, a foundation model specialized for TV navigation. TVTheseus achieves a success rate of $68.3\%$ on TVWorld-N, surpassing strong closed-source baselines such as Gemini 3 Flash and establishing state-of-the-art (SOTA) performance. Additional analyses further provide valuable insights into the development of effective TV-use agents.

Figures (10)

• Figure 1: Illustration of focus-based remote-control TV interaction: discrete key presses (e.g. LEFT/UP/OK) move a highlight across UI elements, inducing UI state transitions toward the target. This process can be formulated as path finding on a topology graph whose nodes are UI states and edges correspond to key-induced transitions.
• Figure 2: Overview of the TVWorld graphs collection pipeline. We perform BFS exploration on physical TV devices (TCL TV and Google TV) to construct initial UI-state graphs, while recording screenshots and view-tree metadata. Graphs are then refined through automated consistency checks and human inspection, producing finalized graphs together with the offline interactive environment for evaluation/training and grounding data.
• Figure 3: Overview of topology-aware training for TVTheseus. Stage I uses topology-priming SFT by distilling topology-aware behaviors and injecting them into the base model using three trace types: geodesic guidance, detour reflection, and stagnation escape. Stage II then applies topology-augmented RL with trace-specific rewards that promote goal-directed progress while discouraging detours and stagnation; example traces appear at the bottom.
• Figure 4: Model performance on TVWorld-N under different visual token budgets.
• Figure 5: Model performance on TVWorld-N under different numbers of historical screenshots.