GRAZE: Grounded Refinement and Motion-Aware Zero-Shot Event Localization

Abstract

American football practice generates video at scale, yet the interaction of interest occupies only a brief window of each long, untrimmed clip. Reliable biomechanical analysis, therefore, depends on spatiotemporal localization that identifies both the interacting entities and the onset of contact. We study First Point of Contact (FPOC), defined as the first frame in which a player physically touches a tackle dummy, in unconstrained practice footage with camera motion, clutter, multiple similarly equipped athletes, and rapid pose changes around impact. We present GRAZE, a training-free pipeline for FPOC localization that requires no labeled tackle-contact examples. GRAZE uses Grounding DINO to discover candidate player-dummy interactions, refines them with motion-aware temporal reasoning, and uses SAM2 as an explicit pixel-level verifier of contact rather than relying on detection confidence alone. This separation between candidate discovery and contact confirmation makes the approach robust to cluttered scenes and unstable grounding near impact. On 738 tackle-practice videos, GRAZE produces valid outputs for 97.4% of clips and localizes FPOC within $\pm$ 10 frames on 77.5% of all clips and within $\pm$ 20 frames on 82.7% of all clips. These results show that frame-accurate contact onset localization in real-world practice footage is feasible without task-specific training.

Figures (5)

• Figure 1: Representative recording challenges in tackle-practice videos. (a) Variable scene composition with multiple players, no visible tackle dummy, and a non-sprinting actor under unstable camera motion. (b) Multiple players with mutual occlusion near the contact window. (c) Player and dummy partially or fully absent due to abrupt panning. (d) Inverted camera orientation with motion blur and multi-player overlap onto the dummy.
• Figure 2: Four-phase pipeline. (1) Grounding: Grounding DINO searches multiple temporal positions with hierarchical prompts and progressive threshold relaxation, retaining all valid player-dummy candidates. (2) Validation: Each candidate is scored by temporal consistency, displacement magnitude, and directional approach toward the dummy; candidates are ranked before SAM2 evaluation. (3) Refinement and Segmentation: Backward refinement recovers $t_{\text{FFBO}}$; SAM2 propagates player and dummy masks from that frame. (4) Contact Verification: FPOC is the first frame where the propagated masks overlap. If no overlap is found, the next ranked candidate is evaluated.
• Figure 3: Temporal localization of first contact using multiframe grounding and backward refinement. (1) Multi-frame grounding queries the untrimmed clip at a set of normalized time positions; in this example, a valid player-dummy pair is first detected at position 0.9, defining a tackle event window. (2) Backward refinement then steps backward from the grounding frame to find FFBO (the first frame in which both the player and dummy are simultaneously present). (3) Starting from FFBO, SAM2 propagates masks forward and identifies FPOC as the earliest frame where the propagated masks provide evidence of player-dummy contact.
• Figure 4: Representative tackle-event frames. Columns show (a) $t_{\text{FFBO}}$ (first frame with both objects visible), (b) $t_{\text{FPOC}}$ (contact onset), and (c) a post-contact frame near $t_{\text{end}}$. Rows show the raw frame, the dummy mask, the player mask, and the composite overlay.
• Figure 5: Ablation study across three performance axes. (a) Segmentation coverage on all 738 videos. Solid bars show correctly segmented clips; hatched bars show clips routed to manual review; the dashed boundary marks the evaluable subset. (b) End-to-end FPOC accuracy at four frame-error tolerances over all 738 clips. Dotted lines mark the best-baseline value per tolerance. (c) Conditional FPOC precision over the evaluable subset. At $\pm5$ frames SOLE (80.3%) and MARS (80.4%) marginally lead GRAZE (79.1%); the full-system advantage emerges and widens from $\pm10$ frames onward.