MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks

TL;DR

The paper tackles the difficulty of solving complex programming tasks with large language models by addressing the tendency to produce monolithic code. It introduces Module-of-Thought (MoT) Instruction Transformation and MoT Instruction Tuning to teach LLMs to decompose problems into modular sub-tasks and implement sub-modules that are then integrated into a final solution. Empirical results on APPS and CodeContests show substantial pass@1 gains (approximately 5.8% on APPS and 5.9% on CodeContests) and enhanced self-correction capabilities (about +3.3%), with further improvements in maintainability of the generated code. Additional analyses reveal that problem complexity benefits from finer modular decomposition, and MoT reduces memory usage while producing more maintainable code, suggesting strong practical value for long-term code maintenance and evolution.

Abstract

Large Language Models (LLMs) have showcased impressive capabilities in handling straightforward programming tasks. However, their performance tends to falter when confronted with more challenging programming problems. We observe that conventional models often generate solutions as monolithic code blocks, restricting their effectiveness in tackling intricate questions. To overcome this limitation, we present Module-of-Thought Coder (MoTCoder). We introduce a framework for MoT instruction tuning, designed to promote the decomposition of tasks into logical sub-tasks and sub-modules. Our investigations reveal that, through the cultivation and utilization of sub-modules, MoTCoder significantly improves both the modularity and correctness of the generated solutions, leading to substantial pass@1 improvements of 5.9% on APPS and 5.8% on CodeContests. MoTCoder also achieved significant improvements in self-correction capabilities, surpassing the current SOTA by 3.3%. Additionally, we provide an analysis of between problem complexity and optimal module decomposition and evaluate the maintainability index, confirming that the code generated by MoTCoder is easier to understand and modify, which can be beneficial for long-term code maintenance and evolution. Our codes are available at https://github.com/dvlab-research/MoTCoder.

Figures (4)

• Figure 1: Pass@1 results on CodeContests (x-axis) and APPS (y-axis). Comparison of our MoTCoder with previous SOTAs. Model size are indicated by scatter size.
• Figure 2: Left: To transform conventional instructions into module-of-thought instructions, our MoT instruction transformation approach guides the models through a two-step process. Initially, the models are instructed to outline the necessary sub-modules, generating only their function headers and docstrings that describe their intended usage. The subsequent instruction then guides the model to implement these sub-modules and integrate them into a cohesive final solution. This instruction is complemented by a one-shot example to further enhance understanding. Right: Examples of instruction, modularized code and solution.
• Figure 3: Illustration of our 2-step Module-of-Thought Instruction Transformation: Plain code utilizes a single module to directly generate code. In contrast, our Module-of-Thought Instruction Transformation first instructs LLMs to outline necessary sub-modules, generating only their function headers and docstrings that describe their intended usage. Subsequently, the instructions guide the model in implementing these sub-modules and eventually combining them into a comprehensive final solution.
• Figure 4: (a) Relationship between the number of functions and accuracy for solutions generated by MoTCoder across different difficulty levels in the APPS dataset. (b) Average time and memory consumption for the passed output for MoT (pink) and Normal (blue). (c) Average maintainability index for MoT finetuning, normal finetuning and baseline models. Valid and test are from CodeContests. Interview, introductory and competition are from APPS test set.