Compressed-Language Models for Understanding Compressed File Formats: a JPEG Exploration

TL;DR

This work explores whether Compressed-Language Models can understand data encoded by Compressed File Formats by training a decoder-only Transformer directly on raw JPEG byte streams. The authors introduce a framework to pre-train CLMs on CFF-encoded files and evaluate them on three tasks: recognizing inherent file properties, handling anomalous files, and generating new JPEGs. Empirical results on MNIST and CIFAR-10 show high JPEG-quality recognition, strong anomaly detection and correction performance, and generation of largely valid JPEGs with correct quality, all without decompression. The findings suggest potential for efficient, compression-native AI systems that operate on raw compressed data, enabling ubiquity-aware and segment-level processing of digital media.

Abstract

This study investigates whether Compressed-Language Models (CLMs), i.e. language models operating on raw byte streams from Compressed File Formats~(CFFs), can understand files compressed by CFFs. We focus on the JPEG format as a representative CFF, given its commonality and its representativeness of key concepts in compression, such as entropy coding and run-length encoding. We test if CLMs understand the JPEG format by probing their capabilities to perform along three axes: recognition of inherent file properties, handling of files with anomalies, and generation of new files. Our findings demonstrate that CLMs can effectively perform these tasks. These results suggest that CLMs can understand the semantics of compressed data when directly operating on the byte streams of files produced by CFFs. The possibility to directly operate on raw compressed files offers the promise to leverage some of their remarkable characteristics, such as their ubiquity, compactness, multi-modality and segment-nature.

Figures (4)

• Figure 1: Testing the understanding capacity of "Compressed-Language Models" (CLMs), i.e. language models trained for next-token prediction on byte streams produced by Compressed File Formats (CFFs). We use the JPEG format as a case study for CFFs. We test the CLMs' understanding of JPEG files by probing their capacity to generate new files, recognize real files, and handle files with anomalies (not displayed in the figure). Our evidence suggests that CLMs can understand JPEG.
• Figure 2: Histograms of log-likelihood differences, $\mathcal{L}$, for MNIST and CIFAR. Positive values dominate, indicating that the model consistently assigns higher likelihoods to natural files compared to their perturbed counterparts. This fact allows the model to correctly tag anomalous files.
• Figure 3: Qualitative results of file generation. We sample files from the model, via greedy decoding, and use the JPEG decoder to obtain an image raster. Here we report images of various JPEG qualities (30, 70, 75, and 85) and all semantic classes from both MNIST (first two rows) and CIFAR (bottom two rows). The bottom row of each dataset displays one sample that was recognized by OpenCV to be a corrupt JPEG. The offending samples are noticeable by their block-like artifacts (MNIST: quality 85, class "2"; CIFAR: quality 75, class "1").
• Figure 4: Qualitative results of file generation via beam search. We sample files from the model via beam search and use the JPEG decoder to obtain an image raster.