Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To See or To Read: User Behavior Reasoning in Multimodal LLMs

Tianning Dong, Luyi Ma, Varun Vasudevan, Jason Cho, Sushant Kumar, Kannan Achan

TL;DR

BehaviorLens investigates how the representation of sequential user history affects reasoning in multimodal LLMs for next-purchase prediction. It compares three modalities—text transcripts, scatter-plot images, and flowchart images—across six MLLMs, measuring accuracy, efficiency, and explanation quality. The key finding is that image-based representations substantially boost prediction accuracy (up to 87.5% improvement) without added computational cost, with model-dependent preferences for scatterplot vs flowchart. The work provides a reproducible benchmarking framework and insights into input design for efficient multimodal reasoning with sequential data.

Abstract

Multimodal Large Language Models (MLLMs) are reshaping how modern agentic systems reason over sequential user-behavior data. However, whether textual or image representations of user behavior data are more effective for maximizing MLLM performance remains underexplored. We present \texttt{BehaviorLens}, a systematic benchmarking framework for assessing modality trade-offs in user-behavior reasoning across six MLLMs by representing transaction data as (1) a text paragraph, (2) a scatter plot, and (3) a flowchart. Using a real-world purchase-sequence dataset, we find that when data is represented as images, MLLMs next-purchase prediction accuracy is improved by 87.5% compared with an equivalent textual representation without any additional computational cost.

To See or To Read: User Behavior Reasoning in Multimodal LLMs

TL;DR

BehaviorLens investigates how the representation of sequential user history affects reasoning in multimodal LLMs for next-purchase prediction. It compares three modalities—text transcripts, scatter-plot images, and flowchart images—across six MLLMs, measuring accuracy, efficiency, and explanation quality. The key finding is that image-based representations substantially boost prediction accuracy (up to 87.5% improvement) without added computational cost, with model-dependent preferences for scatterplot vs flowchart. The work provides a reproducible benchmarking framework and insights into input design for efficient multimodal reasoning with sequential data.

Abstract

Multimodal Large Language Models (MLLMs) are reshaping how modern agentic systems reason over sequential user-behavior data. However, whether textual or image representations of user behavior data are more effective for maximizing MLLM performance remains underexplored. We present \texttt{BehaviorLens}, a systematic benchmarking framework for assessing modality trade-offs in user-behavior reasoning across six MLLMs by representing transaction data as (1) a text paragraph, (2) a scatter plot, and (3) a flowchart. Using a real-world purchase-sequence dataset, we find that when data is represented as images, MLLMs next-purchase prediction accuracy is improved by 87.5% compared with an equivalent textual representation without any additional computational cost.

Paper Structure

This paper contains 17 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. User Behavior Representations
  4. User Intent Prediction and Reasoning
  5. Experiments
  6. Results
  7. Case Study
  8. Conclusion
  9. Dataset Preparation and Metrics
  10. Dataset
  11. Metrics and Evaluation
  12. Prompts for User Behavior Reasoning
  13. Multi-dimensional Evaluation of Explanation
  14. Related Works
  15. Next Item Prediction
  16. ...and 2 more sections

Figures (4)

  • Figure 1: A schematic showing different representations (text, scatter plot, and flow chart) of user history for next purchase prediction using multimodal LLMs.
  • Figure 2: Comparison in MMLM explanations. Each metric is measured by a scale of 1 to 5. Higher score indicates better reasoning in given metric. Blue dots are reasoning of text input. Green dots are reasoning of scatterplot input. Red dots are reasoning of flowchart input.
  • Figure 3: Flowchart of user history
  • Figure 4: Scatterplot of user history