A comprehensive survey by 32+ tokenizer researchers covering subword algorithms, multilingual pitfalls, evaluation, theory, security, and alternatives—framing tokenization as a first-class bottleneck in the LLM era.
Tokenization: A Survey for Modern NLP
Marco Cognetta, Christopher Akiki, Pawan Sasanka Ammanamanchi, Catherine Arnett, Thomas Bauwens, and many co-authors (32+ tokenizer researchers) — submitted 30 Sept 2026 — alphaXiv
Abstract
While modern language models take raw text as their input and produce raw text as output, they do not operate over text directly. Hidden in the very first step of language model pipelines is tokenization, where raw text is converted to a sequence of tokens which the model is designed to process.
The dominant approach to tokenization is subword tokenization, which uses tokens whose granularity lies between that of characters (or bytes) and words. Subword tokenization is ubiquitous in modern natural language processing (NLP) due to its advantages: compared to word-level models, it solves the out-of-vocabulary problem while constraining the vocabulary size, reducing the number of embedding parameters and logits needed. Compared to character-level models, it improves the inductive bias of models by packing more information into a token while producing significantly shorter sequences.
While this technique is powerful, it is not without flaws. To name a few: it imposes a rigid system for how tokens are built, and one system may not generalize across languages; segmentation ambiguities create room for adversarial attacks; in multilingual settings, balancing the token allocation between high- and low-resource languages must be done explicitly; and the reliance on a static, pre-computed vocabulary means the tokenization process is disjoint from the model's training, preventing true end-to-end optimization. There is substantial room for innovation in this domain, yet progress is contingent upon a clear understanding of existing methodologies and current bottlenecks. This survey aims to serve that purpose.
We present tokenization in the context of the modern large language model (LLM) era and cover the breadth of areas that tokenization directly influences. We discuss algorithms, techniques, pitfalls, theory, and practice, along with jumping-off points for prospective researchers and practitioners. We hope that this survey brings more attention to the field of tokenization as an area in which much progress is still to be made.
Scope (from authors’ announcement)
The survey covers algorithms, evaluations, multilinguality, encodings, theory, constrained generation, token healing, tokenizer security concerns, and alternatives (e.g., latent or visual tokenization). Sections include questions for prospective researchers and practitioners.