ShIOEnv: A CLI Behavior-Capturing Environment Enabling Grammar-Guided Command Synthesis for Dataset Curation
Jarrod Ragsdale, Rajendra Boppana
TL;DR
ShIOEnv addresses the lack of execution-grounded CLI datasets by modeling command construction as a Markov decision process and constraining the action space with context-free grammars derived from man pages. A grammar-constrained PPO policy is trained to maximize a dense redundancy-based reward that favors minimally sufficient, behaviorally distinctive command sequences, producing a large, execution-grounded dataset (ShIOEnv-40c) for 40 utilities. Fine-tuning CodeT5 on these data yields substantial gains in generation fidelity (BLEU-4 and chrF) and demonstrates the value of grammar-guided exploration for dataset curation, especially when comparing constrained versus unconstrained policies. The work provides a reusable, extendable environment for CLI-behavior modeling and suggests directions for curriculum learning, multi-command planning, and broader command coverage in future research.
Abstract
Command-line interfaces (CLIs) provide structured textual environments for system administration. Explorations have been performed using pre-trained language models (PLMs) to simulate these environments for safe interaction in high-risk environments. However, their use has been constrained to frozen, large parameter models like GPT. For smaller architectures to reach a similar level of believability, a rich dataset of CLI interactions is required. Existing public datasets focus on mapping natural-language tasks to commands, omitting crucial execution data such as exit codes, outputs, and environmental side effects, limiting their usability for behavioral modeling. We introduce a Shell Input -Output Environment (ShIOEnv), which casts command construction as a Markov Decision Process whose state is the partially built sequence and whose actions append arguments. After each action, ShIOEnv executes the candidate and returns its exit status, output, and progress toward a minimal-length behavioral objective. Due to the intractable nature of the combinatorial argument state-action space, we derive a context-free grammar from man pages to mask invalid arguments from being emitted. We explore random and proximal-policy optimization (PPO)-optimized sampling of unrestricted and grammar-masked action spaces to produce four exploration strategies. We observed that grammar masking and PPO significantly improve sample efficiency to produce a higher quality dataset (maximizing the number of arguments while minimizing redundancies). Policy-generated datasets of shell input-output behavior pairs are used to fine-tune CodeT5, where we observe 85% improvements in BLEU-4 when constraining the action space to grammar productions with an additional 26% improvement when applying PPO. The ShIOEnv environment and curated command behavior datasets are released for use in future research.