Task Me Anything

TL;DR

Task-Me-Anything introduces a scalable, on-demand benchmark generation framework for large multimodal language models that moves beyond general-purpose benchmarks to task-specific evaluation. It builds an extendable spatio-temporal scene-graph taxonomy, 28 task generators, and large synthetic/real data sources to produce up to 750M VQA tasks, with a budget-aware evaluation pipeline and a UI for interactive analysis. The authors provide extensive open-source MLM evaluations, revealing that open-source models excel in object/attribute recognition but struggle with spatial and temporal understanding, while larger and proprietary models generally perform better but with notable exceptions; GPT4o, in particular, shows weaknesses in rotating/moving objects and color distinctions. A 2024 benchmark variant further emphasizes the most challenging tasks, validating the framework’s ability to identify current model shortcomings under budget constraints. Overall, Task-Me-Anything offers a practical, query-driven toolset for model selection, debugging, and targeted improvement, with strong implications for deployment in real-world applications and ongoing MLM research.

Abstract

Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.

Figures (41)

• Figure 1: We present examples of user queries, corresponding task instances generated by Task-Me-Anything as well as the evaluation results on them that answer the queries.
• Figure 2: We present the key components in Task-Me-Anything. The top part illustrates the task generation process with an example video synthesized with 3D objects and their annotations, and the task generator for generating questions about rotating objects' attributes. The bottom part depicts the model evaluation process, which selects the relevant tasks based on the user's query and their budget and performs either full evaluation or results approximation to answer the query.
• Figure 3: The statistics of generatable tasks of each task generator and example image/video in Task-Me-Anything. We each task generator with high-level perceptual skills and this collection of task generators can collectively generate over 750M VQA tasks.
• Figure 4: We adopt two distinct prompts, the detailed prompt and the succinct prompt, in our evaluation to assess models' sensitivity to different prompts.
• Figure 5: Task-Me-Anything-UI Interface.