Trim My View: An LLM-Based Code Query System for Module Retrieval in Robotic Firmware

TL;DR

This work addresses the difficulty of understanding stripped binary firmware by combining binary modularization, decompilation, and LLM-based function summarization into ChatCPS, a code query system for module retrieval in robotic firmware. The approach uses an ARM-adapted BCD decomposition to form modules, a two-pass summarization strategy with three open-source LLMs, and a category-retrieval step aided by tailored prompts, achieving a best average module-categorization F1 score of $0.68$ on ArduPilot across 467 modules. Key contributions include the ARM-focused modification of BCD, empirical evaluation of three open-source LLMs for function summarization and module retrieval, and ground-truth establishment for CPS-relevant categories (data transfer, navigation, control, safety checks, and other). The results demonstrate that CodeStral consistently outperforms peers in module retrieval while enabling efficient, open-source-based analysis, with practical implications for rapid firmware comprehension and targeted reverse-engineering workflows in robotics.

Abstract

The software compilation process has a tendency to obscure the original design of the system and makes it difficult both to identify individual components and discern their purpose simply by examining the resulting binary code. Although decompilation techniques attempt to recover higher-level source code from the machine code in question, they are not fully able to restore the semantics of the original functions. Furthermore, binaries are often stripped of metadata, and this makes it challenging to reverse engineer complex binary software. In this paper we show how a combination of binary decomposition techniques, decompilation passes, and LLM-powered function summarization can be used to build an economical engine to identify modules in stripped binaries and associate them with high-level natural language descriptions. We instantiated this technique with three underlying open-source LLMs -- CodeQwen, DeepSeek-Coder and CodeStral -- and measured its effectiveness in identifying modules in robotics firmware. This experimental evaluation involved 467 modules from four devices from the ArduPilot software suite, and showed that CodeStral, the best-performing backend LLM, achieves an average F1-score of 0.68 with an online running time of just a handful of seconds.

Figures (5)

• Figure 1: The Workflow of the ChatCPS system. A) ChatCPS decomposes the binary firmware into its modules. B) Shows the decompilation process. $F_n$ indicates binary functions, while $F_n^*$ indicates the corresponding decompiled functions. C) ChatCPS summarizes each function in a module using three open-source LLMs (CodeQwen, CodeStral, DeepSeek-Coder) and D) categorizes modules based on function descriptions.
• Figure 2: Illustration of how the BCD algorithm uses information about the call graph, data references, and code locality to decompose the binary into groups of functions. $F_n$ indicates functions in the binary. The image shows a final weighted graph generated by the BCD algorithm. Green edges show the function call relationship between functions. Red edges show the code locality, and the purple edges indicate the data references.
• Figure 3: The process of summarizing decompiled functions using LLMs. Section A-- presents a decompiled example of the function _mav_finalize_message_chan_send from the ArduPilot project, extracted from the stripped binary of a copter device compiled for the fmuv3 board. Section B-- depicts the prompt used to summarize decompiled functions. Section C-- displays the function descriptions of _mav_finalize_message_chan_send, as returned by three distinct LLMs.
• Figure 4: Module Retrieval Process. We provide function summaries and module categories as an input prompt to an LLM. Then, we instruct the LLM to decide based on the function descriptions and ranked the categories.
• Figure 5: Comparison of time required for function summarization by three open-source LLMs (CodeStral, CodeQwen, and DeepSeek-Coder) across four ArduPilot devices.