MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark

TL;DR

The paper tackles the problem of evaluating instruction following in multimodal, multi-turn dialogues, where instructions are dispersed across a long chat context and must be verified via execution. It introduces MMMT-IF, an image-based benchmark extending the MMDU dataset with global, trackable instructions and two objective metrics, PIF and PIF-N-K, that quantify instruction adherence and robustness. Empirical results reveal that model performance degrades as the number of instructions grows and that retrieval of instructions from the conversation context is a major bottleneck; appending all instructions to the end of the input markedly improves performance, underscoring the retrieval challenge. The work also provides human and autorater evaluation schemas, analyzes model biases in LLM judges, and offers paths toward open-source resources and future directions such as reinforcement learning from execution feedback and dependent instruction extensions.

Abstract

Evaluating instruction following capabilities for multimodal, multi-turn dialogue is challenging. With potentially multiple instructions in the input model context, the task is time-consuming for human raters and we show LLM based judges are biased towards answers from the same model. We propose MMMT-IF, an image based multi-turn Q$\&$A evaluation set with added global instructions between questions, constraining the answer format. This challenges models to retrieve instructions dispersed across long dialogues and reason under instruction constraints. All instructions are objectively verifiable through code execution. We introduce the Programmatic Instruction Following ($\operatorname{PIF}$) metric to measure the fraction of the instructions that are correctly followed while performing a reasoning task. The $\operatorname{PIF-N-K}$ set of metrics further evaluates robustness by measuring the fraction of samples in a corpus where, for each sample, at least K out of N generated model responses achieve a $\operatorname{PIF}$ score of one. The $\operatorname{PIF}$ metric aligns with human instruction following ratings, showing 60 percent correlation. Experiments show Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet, have a $\operatorname{PIF}$ metric that drops from 0.81 on average at turn 1 across the models, to 0.64 at turn 20. Across all turns, when each response is repeated 4 times ($\operatorname{PIF-4-4}$), GPT-4o and Gemini successfully follow all instructions only $11\%$ of the time. When all the instructions are also appended to the end of the model input context, the $\operatorname{PIF}$ metric improves by 22.3 points on average, showing that the challenge with the task lies not only in following the instructions, but also in retrieving the instructions spread out in the model context. We plan to open source the MMMT-IF dataset and metric computation code.

Figures (16)

• Figure 1: The number of turns of all chats in the evaluation dataset.
• Figure 2: For all 990 turns, the distribution of the number of instructions that were given so far in the chat.
• Figure 3: The distribution of the input context lengths for Gemini 1.5 Pro, Claude 3.5 Sonnet and GPT-4o, in the evaluation dataset, along with the mean input context length in characters.
• Figure 4: The image above shows the $\operatorname{PIF}$ metric conditioned on the question turn with 95% confidence intervals. For a fixed turn $i$, the mean is taken across all chats at with at least $i$ turns.
• Figure 5: The mean programmatic instruction following score conditional on the number of instructions given in the chat so far. The metric defaults to 1 if no instruction has been given.