X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning

TL;DR

This work introduces X-Ego-CS, the first cross-egocentric multi-agent esports dataset, and proposes Cross-Ego Contrastive Learning (CECL) to align teammates' first-person video streams at the same timestep. CECL promotes a shared team-state representation, enabling an individual agent to infer both teammate and opponent locations from a single egocentric view, especially under partial observability. The approach is evaluated on a teammate-opponent location prediction task across multiple video encoders, showing consistent improvements in low-POV scenarios and revealing trade-offs at full team visibility. Collectively, the dataset and method establish a foundation for cross-egocentric multi-agent benchmarking and advance spatiotemporal reasoning and human-AI teaming in complex, real-time environments.

Abstract

Human team tactics emerge from each player's individual perspective and their ability to anticipate, interpret, and adapt to teammates' intentions. While advances in video understanding have improved the modeling of team interactions in sports, most existing work relies on third-person broadcast views and overlooks the synchronous, egocentric nature of multi-agent learning. We introduce X-Ego-CS, a benchmark dataset consisting of 124 hours of gameplay footage from 45 professional-level matches of the popular e-sports game Counter-Strike 2, designed to facilitate research on multi-agent decision-making in complex 3D environments. X-Ego-CS provides cross-egocentric video streams that synchronously capture all players' first-person perspectives along with state-action trajectories. Building on this resource, we propose Cross-Ego Contrastive Learning (CECL), which aligns teammates' egocentric visual streams to foster team-level tactical situational awareness from an individual's perspective. We evaluate CECL on a teammate-opponent location prediction task, demonstrating its effectiveness in enhancing an agent's ability to infer both teammate and opponent positions from a single first-person view using state-of-the-art video encoders. Together, X-Ego-CS and CECL establish a foundation for cross-egocentric multi-agent benchmarking in esports. More broadly, our work positions gameplay understanding as a testbed for multi-agent modeling and tactical learning, with implications for spatiotemporal reasoning and human-AI teaming in both virtual and real-world domains. Code and dataset are available at https://github.com/HATS-ICT/x-ego.

Figures (5)

• Figure 1: Illustration of Cross-Ego Contrastive Learning (CECL), where teammates' egocentric video representations are aligned across teams.
• Figure 2: Additional statistics of the X-Ego-CS dataset. (a) Player FACEIT Elo Rating Distribution: above 2000 corresponds to the highest FACEIT level 10 rank (roughly top 10% of players), and above 3000 represents roughly the top 1%. (b) Match Duration. (c) Player Alive Time Duration, which corresponds to the duration of video files. (d) Player Trajectory Heatmap aggregated from both team sides across all matches, visualized on the de_mirage map.
• Figure 3: Illustration of the forward pass on a data batch containing three teams of three agents each. (a) Agents from the same team are treated as positive pairs, while agents from different teams or time segments are treated as negatives. (b) Each agent's egocentric video segment is encoded into a representation vector; a mask is applied to the top-left corner to prevent leaking teammate or opponent locations from the mini-map. (c) The CECL objective aligns teammates' egocentric video representations within the same batch while repelling others. A subset of agents from a team is then selected for team-level location prediction, formulated as a multi-label classification task.
• Figure 4: Performance comparison of CECL vs. baseline across different visual encoders (DINOv2, ViViT, VideoMAE, SigLIP) as the number of agent POVs varies from 1 to full team size of 5. CECL shows substantial gains at lower POV counts (1-2 agents) but diminishing returns as team coverage increases.
• Figure 5: t-SNE visualization of learned embeddings before (bottom row) and after (top row) cross-egocentric contrastive learning. Each point represents a video clip embedding. Columns show different coloring schemes: location (left), time (middle), and team (right). After contrastive training, embeddings demonstrate substantially improved clustering with 8.87% increase in location-based separation and 15.86% increase in team-based separation, while maintaining smooth temporal gradients.