Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility

TL;DR

RTCE introduces a lossless round-trip benchmark for code understanding in Code-LLMs by enforcing a bijective encoding/decoding pipeline with the strict constraint $dec(enc(x)) = x$. It constructs a deterministic dataset using four classic compression algorithms (LZW, Arithmetic Encoding, Run-Length Encoding, Huffman) across four data families, totaling 250 inputs and 1000 evaluation instances. The authors evaluate zero-shot, supervised trace-based fine-tuning, and self-reflection, finding modest improvements but no model achieving full round-trip consistency, indicating gaps in internal coherence for bidirectional code reasoning. They argue that round-trip bijection exposes deeper mechanistic understanding than forward-only benchmarks and release the code and data for community use.

Abstract

LLMs demonstrate strong performance on code benchmarks, yet round-trip code execution reveals limitations in their ability to maintain consistent reasoning across forward and backward execution. We present RoundTripCodeEval (RTCE), a comprehensive benchmark consisting of four distinct code execution reasoning tasks designed to rigorously test round-trip consistency. RTCE provides an execution-free, exact-match evaluation of bijection fidelity, assessing whether models preserve a consistent one-to-one mapping between encoding and decoding operations across various algorithms and directions. We systematically evaluate state-of-the-art Code-LLMs using zero-shot prompting, supervised fine-tuning on execution traces, and self-reflection mechanisms. Each yields modest improvements, but none closes the gap, indicating that current LLMs struggle with true round-trip consistency, which demonstrates that they lack the internal coherence required for trustworthy code reasoning. RTCE surfaces several new and previously unmeasured insights that are not captured by existing I/O-prediction, execution-reasoning, or round-trip natural-language benchmarks. We will release the code and the dataset upon acceptance.

Figures (20)

• Figure 1: A standard lossless compression pipeline, where an input $X$ is encoded into a compact representation $z$ and then decoded back into $X'$. Here, enc and dec can be referred to as the encoding and decoding functions. We outline the overall workflow of our pipeline using a concrete example in Appendix \ref{['sec:task-example']}.
• Figure 2: Overview of the reasoning tasks which depict our four-step round-trip procedure for assessing code compression self-consistency. In (1), Output Prediction, the input is transformed using the actual encoder ${\footnotesize X \xrightarrow{\mathsf{enc}} z}$ and the LLM predicts $z$ from $X$ and $\mathsf{enc}$. In (2), Input Prediction with Inversion, the compressed output is mapped back to the input space ${\footnotesize z \xrightarrow{\mathsf{inv\_enc}} X'}$ using the inverted encoder $\mathsf{inv\_enc}$ to reconstruct $X'$. In (3), Output Prediction with Inversion, the input is passed through the inverted decoder ${\footnotesize X \xrightarrow{\mathsf{inv\_dec}} z}$ to recover $z$. In (4), Input Prediction, the compressed output is decoded ${\footnotesize z \xrightarrow{\mathsf{dec}} X'}$ using the actual decoder $\mathsf{dec}$ to produce the final reconstruction $X'$. A round-trip check ${\footnotesize X {=} X'}$ verifies perfect reconstruction fidelity, returning a binary result that indicates if the LLM is self-consistent or not. This corresponds to a chain of length 1, but this could be extended to an arbitrary length.
• Figure 3: Multi-turn revision for AE on a subset of the dataset. Overall, tasks requiring inversion show lower initial accuracy but larger gains with revision across the two rounds.
• Figure 4: A concrete task example outlining the workflow across the four prediction tasks using run-length encoding.
• Figure 5: Input prediction prompt template for RLE algorithm.