SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models

TL;DR

This work tackles efficient boundary placement in end-to-end tokenizer-free hierarchical sequence models by introducing a router-agnostic boundary quality metric, boundary enrichment $B$, which gauges whether chunk starts occur at positions with high next-byte surprisal. It then proposes Sombrero, a Hierarchical Autoregressive Transformer variant that steers boundary placement toward predictive difficulty using a confidence-alignment boundary (CAB) loss and stabilizes learning with byte-level smoothing, rather than chunk-level smoothing. Empirically, on 1B-scale UTF-8 data spanning English, German, code, and math, Sombrero improves the accuracy–efficiency trade-off and yields boundaries that better align compute with hard-to-predict positions, with substantial gains in boundary enrichment and robust target compression across settings. The results support tokenizer-free hierarchical modeling as a practical alternative to fixed-tokenization schemes and provide concrete diagnostics and training levers for steering where expensive backbone compute is spent, while outlining directions for scaling, deeper hierarchies, and hardware-aware optimizations.

Abstract

Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end methods can learn meaningful boundaries from the language-modeling objective alone, it remains difficult to quantitatively assess and systematically steer where compute is spent. We introduce a router-agnostic metric of boundary quality, boundary enrichment B, which measures how strongly chunk starts concentrate on positions with high next-byte surprisal. Guided by this metric, we propose Sombrero, which steers boundary placement toward predictive difficulty via a confidence-alignment boundary loss and stabilizes boundary learning by applying confidence-weighted smoothing at the input level rather than on realized chunks. On 1B scale, across UTF-8 corpora covering English and German text as well as code and mathematical content, Sombrero improves the accuracy-efficiency trade-off and yields boundaries that more consistently align compute with hard-to-predict positions.

Figures (4)

• Figure 1: Illustration of the HAT architecture and information flow. Arrows in red indicate connections along which gradient affecting the boundaries $\mathbf{b}$ flows backward. Green boxes indicate the changes contributed in Sombrero: a simplified sigmoid chunker, byte-level temporal expansion, and an additional confidence-alignment boundary loss.
• Figure 2: We compare chunk- and byte-level smoothing in combination with cosine and sigmoid boundary scores on a synthetic problem (see Section \ref{['appendix:synthetic_problem']}). Sigmoid boundary scores underperform with chunk-level smoothing but excel with byte-level smoothing.
• Figure 3: Empirical compression $C_{emp}$ versus bits-per-byte when varying target compression $C_{tar}$ for 1B Sombrero and H-Net.
• Figure 4: Training curves of selected metrics for different configurations of the design ladder. BPB$_0$ indicates the bits-per-byte averaged only over the first bytes of chunks, where higher indicates boundaries on more difficult bytes.