EffGen: Enabling Small Language Models as Capable Autonomous Agents

TL;DR

EffGen addresses the practicality gap of agentic AI by tailoring an agent framework for small language models that can run locally. It fuses model-size-aware prompt optimization, pre-execution complexity routing, intelligent task decomposition, and a three-tier memory system to deliver competitive task success, reduced token usage, and secure deployment. The framework unifies three major agent protocols (MCP, A2A, ACP) and demonstrates strong gains across 13 benchmarks, with particularly large improvements for small models and substantial code-task benefits from structured tool usage. Its open-source MIT license and cross-model/back-end compatibility position EffGen as a practical platform for research, development, and deployment of efficient autonomous agents. The results reveal complementary scaling: prompt optimization benefits SLMs most at small scales, while complexity routing yields larger gains for bigger models, suggesting continued co-design of models and frameworks for optimal performance.

Abstract

Most existing language model agentic systems today are built and optimized for large language models (e.g., GPT, Claude, Gemini) via API calls. While powerful, this approach faces several limitations including high token costs and privacy concerns for sensitive applications. We introduce effGen, an open-source agentic framework optimized for small language models (SLMs) that enables effective, efficient, and secure local deployment (pip install effgen). effGen makes four major contributions: (1) Enhanced tool-calling with prompt optimization that compresses contexts by 70-80% while preserving task semantics, (2) Intelligent task decomposition that breaks complex queries into parallel or sequential subtasks based on dependencies, (3) Complexity-based routing using five factors to make smart pre-execution decisions, and (4) Unified memory system combining short-term, long-term, and vector-based storage. Additionally, effGen unifies multiple agent protocols (MCP, A2A, ACP) for cross-protocol communication. Results on 13 benchmarks show effGen outperforms LangChain, AutoGen, and Smolagents with higher success rates, faster execution, and lower memory. Our results reveal that prompt optimization and complexity routing have complementary scaling behavior: optimization benefits SLMs more (11.2% gain at 1.5B vs 2.4% at 32B), while routing benefits large models more (3.6% at 1.5B vs 7.9% at 32B), providing consistent gains across all scales when combined. effGen (https://effgen.org/) is released under the MIT License, ensuring broad accessibility for research and commercial use. Our framework code is publicly available at https://github.com/ctrl-gaurav/effGen.

Figures (16)

• Figure 1: Overview of the EffGen framework. Given an input query, EffGen first computes a complexity score using five weighted factors. Based on this score, tasks are routed to either single-agent execution (for simple tasks) or multi-agent decomposition (for complex tasks). The prompt optimizer compresses and restructures prompts for small models, while the memory system provides relevant context. This design enables small language models to handle agentic tasks that typically require much larger models.
• Figure 2: Execution time scaling across parameter sizes.EffGen achieves 18$\times$ speedup at 1.5B (vs Smolagents). LangChain exhibits super-linear scaling on Calculator benchmarks (11.2$\times$ time increase for 21.3$\times$ parameter growth), while EffGen maintains near-linear scaling. See Appendix \ref{['app:timing_analysis']}.
• Figure 3: Error analysis on failed cases.
• Figure 4: Complete agent execution pipeline (Algorithm 1) with complexity-based routing. The pipeline begins by computing complexity score $\mathcal{C}(q)$ and optimizing the prompt with context. Simple tasks ($c < \tau$) execute on a single ReAct agent. Complex tasks undergo decomposition $\mathcal{D}(q)$ producing subtasks with dependency graph $G$, then route to one of three execution strategies: parallel (independent subtasks with synthesis), sequential (pipeline where last step provides answer), or hybrid (topological sort of DAG with mixed parallel/sequential execution and synthesis). All paths converge to memory update before returning the final response.
• Figure 5: Prompt optimization pipeline architecture. The system detects model size and applies five sequential transformations: (1) pattern-based compression using 30 validated patterns, (2) sentence simplification by splitting at conjunctions, (3) redundancy removal using Jaccard similarity, (4) bullet formatting for structured instructions, and (5) optional context truncation with priority preservation. The full pipeline achieves 57% token reduction with 8.8% accuracy improvement on benchmark tasks. Bullet formatting provides the largest accuracy gain (+5.3%), while pattern compression provides the largest token reduction (-25%). Effects are largely additive, indicating complementary optimization directions.

Theorems & Definitions (5)

• Definition 3.1: Model Size Categories
• Definition 3.2: Complexity Score
• Theorem K.1: Calculator Safety
• proof
• Definition S.1: Session