PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction

TL;DR

This work tackles predicting the time and location of events from images, a challenging task requiring multi-step, human-like puzzle solving. It introduces PuzzleGPT, a modular pipeline with Perceiver, Reasoner, Combiner, Noise Filter, and Online Retriever, augmented by a confidence-based hierarchical fusion to efficiently combine clues and suppress noise. Empirically, PuzzleGPT achieves state-of-the-art zero-shot performance on TARA and WikiTiLo, outperforming large Vision-Language Models and automatic modular pipelines, with ablations confirming the contributions of retrieval, noise filtering, and hierarchical reasoning. The approach advances practical applications in timeline construction and media analysis by enabling robust, interpretable reasoning across diverse cues, and highlights the benefits of task-specific modular design over automatic pipeline generation.

Abstract

The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models.

Figures (11)

• Figure 1: An illustration of different skills (Puzzle-like) required for solving Time and Reasoning prediction tasks.
• Figure 2: PuzzleGPT overview depicting each of the components -- Perceiver, Reasoner, Combiner, Noise Filter, and the Online Retriever. The modular approach makes PuzzleGPT interpretable and robust. All VLMs/LLMs are pretrained and frozen.
• Figure 3: Extra location information in the label causes even correct prediction to be classified as wrong with exact-match accuracy metric. We mitigate this by label standardization.
• Figure 4: An illustration of the importance of confidence-based hierarchical combination of information. 1st Hierarchy leads to a scarcity of information, while 3rd Hierarchy is noisy, and underscores the need for a confidence-based hierarchical combination.
• Figure 5: Top: Ablation on Hash Threshold (HT): peak performance at 5, with noisy performance on both lower or higher HT. Bottom: Ablation on Retrieval Threshold (RT): retrieval is best at 90, with either side of it leading to noisy retrieval. Loc F1$^\beta$ is Location X-F1$^\beta$. Time F1$^\beta$ is Time X-F1$^\beta$.