TEM Agent: enhancing transmission electron microscopy (TEM) with modern AI tools

TL;DR

The paper addresses the challenge of operating complex transmission electron microscopes and managing multimodal data by introducing TEM Agent, a framework that uses the Model Context Protocol to interface an off-the-shelf LLM with TEM subsystems and data platforms. The method integrates four MCP servers—microscope control, detector control, Crucible metadata, and Distiller data workflows—via a text-based command protocol, avoiding domain-specific training. Key findings show that TEM Agent can perform simple operations, automate tomography workflows, and leverage historical metadata to guide experiments and optimize advanced techniques like ptychography, while preserving safety and requiring human oversight. The work demonstrates a path toward accessible, scalable automation in electron microscopy and suggests future expansions to other instruments and richer image-processing capabilities.

Abstract

Recent improvements in large language models (LLMs) have had a dramatic effect on capabilities and productivity across many disciplines involving critical thinking and writing. The development of the model context protocol (MCP) provides a way to extend the power of LLMs to a specific set of tasks or scientific equipment with help from curated tools and resources. Here, we describe a framework called TEM Agent designed for transmission electron microscopy (TEM) that leverages the benefits of LLMs through a MCP approach. We simultaneously access and control several subsystems of the TEM, a data management platform, and high performance computing resources through text-based instructions. We demonstrate the abilities of the TEM Agent to set up and complete intricate workflows using a simplified set of MCP tools and resources accompanying a commercial LLM without any additional training. The use of a framework such as the TEM Agent simplifies access to complex microscope ecosystems comprised of several vendor and custom systems enhancing the ability of users to accomplish microscopy experiments across a range of difficulty levels.

Figures (4)

• Figure 1: The TEM Agent framework incorporates a commercial LLM and a set of MCP servers with access to various microscope components. The cloud-based LLM was accessed through Claude-code running on a user's laptop. Several MCP servers are run on the internet-connected Microscope Support PC. The tools available to the LLM are shown as colored boxes. One resource describing the microscope and common experiments was also included. The microscope and detector MCP servers communicate with custom zeroMQ servers installed on each equipment control computer over a local area connection. This client/server model is required because equipment control computers are isolated from the internet. The Crucible and Distiller platforms expose an API to the internet and can be accessed from anywhere.
• Figure 2: Chaining together complex experimental workflows through the example of (a) tomography: (b) conventional workflows require many manual steps, while (c) the dynamic MCP workflow is adjustable and requires minimal user support leading to (d-e) high-quality tomographic datasets.
• Figure 3: TEM Agent allows seamless integration of our Crucible database into an experiment. Crucible contains experimental data and metadata that is accessible through an API. TEM Agent can interpret user prompts to identify and apply the filters to find relevant historical datasets and then drieclty apply those settings to the microscope.
• Figure 4: (a) TEM agent queries metadata from Distiller, summarizes the information for the user, (b) and uses the ptychography tool to optimize data acquisition conditions. (c) Our framework results in a high-quality final reconstruction of the experimental data (completed offline).