NLP

Tokenization: A Survey for Modern NLP

A comprehensive survey by 32+ tokenizer researchers covering subword algorithms, multilingual pitfalls, evaluation, theory, security, and alternatives—framing tokenization as a first-class bottleneck in the LLM era.

Research · AI · LLMs · Research · Machine Learning