ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memory

TL;DR

ToolMem introduces a learnable, structured memory of tool capabilities for multimodal agents to cope with uncertain neural-tool outputs. It initializes a proficiency-based memory taxonomy, learns from interaction experiences via a memory-induction module, and updates memories through retrieval-augmented generation. In extensive experiments on text-generation and text-to-image tasks, ToolMem significantly improves tool performance prediction (e.g., MAE/RMSE reductions and higher Pearson correlations) and enhances tool selection accuracy across diverse tool pairs, especially when capability gaps are large. This memory-guided approach enables adaptive, data-driven tool choices without retraining, advancing robust multi-tool reasoning in dynamic generative settings.

Abstract

Agents utilizing tools powered by large language models (LLMs) or vision-language models (VLMs) have demonstrated remarkable progress in diverse tasks across text and visual modalities. Unlike traditional tools such as calculators, which give deterministic outputs, neural tools perform uncertainly across task scenarios. While different tools for a task may excel in varied scenarios, existing agents typically rely on fixed tools, thus limiting the flexibility in selecting the most suitable tool for specific tasks. In contrast, humans snowball their understanding of the capabilities of different tools by interacting with them, and apply this knowledge to select the optimal tool when solving a future task. To build agents that similarly benefit from this process, we propose ToolMem that enables agents to develop memories of tool capabilities from previous interactions, by summarizing their strengths and weaknesses and storing them in memory; at inference, the agent can retrieve relevant entries from ToolMem, and select the best tool to solve individual tasks more accurately. We evaluate ToolMem on learning varied text generation and text-to-image generation neural tools. Compared to no-memory, generic agents, we find ToolMem-augmented agents predict tool performance 14.8% and 28.7% more accurately across text and multimodal generation scenarios. Moreover, ToolMem facilitates optimal tool selection among multiple choices by 21% and 24% absolute increases in respective scenarios.

Figures (4)

• Figure 1: Example of identifying better-performing tools based on learned ToolMem. For example, compared to the image generation tool instantiated by Midjourney, SDXL-Base tool is better at rendering short text such as the queried "Eco Market". We thus store this information in ToolMem and prioritize selecting SDXL-base in later tasks that ask to render short texts.
• Figure 2: Illustration of the ToolMem construction process from past experiences (left), as well as ToolMem-augmented task-solving and capability analysis at downstream (right).
• Figure 3: ToolMem's score prediction accuracy on six text-to-image models across different top-$k$ memory retrieval sizes, evaluated on 100 samples. The dashed line at $k=12$ indicates most models' empirically optimal memory size.
• Figure 4: Qualitative examples for ToolMem. The left image illustrates the prediction for "An oil painting of a majestic lion hanging above the fireplace," while the right example shows the prediction for "A pair of socks, one is striped, but the other is not."