Tokenization

From Tokens to Bytes I: Scaling Hierarchical Networks

Cartesia Research reports a newer generation of hierarchical networks (H-Nets) that learn directly from raw bytes via dynamic chunking instead of a tokenizer. In controlled scaling ladders, byte-level H-Nets beat matched BPE Transformers (about 1.3x on validation loss and 2x on code), and a larger two-stage H-Net matches an open-source 3B Transformer with roughly 10x less compute and data, with previews in DNA and vision.

Research · AI Research · LLMs · Machine Learning · Tokenization