September 21, 2026
The same code tokenizes 3–30 times faster: tokenizers v1 kept the API and token ID
Hugging Face made tokenization 3–30 times faster across ten model families while keeping the same token IDs: no rewrites required.
On one laptop core, Hugging Face's new tokenizer processes 90 MB of text per second. Version 0.23.1 delivered 8.
That is a tokbench measurement on GPT-2 (Apple M3 Max, one thread): a 10.7x increase. The tiktoken library delivers 27 MB/s on the same run.
For decoding, the new pipeline delivers 337 MB/s, 6.1 times faster than the reference. Only tokie is faster at 379 MB/s.
No rewrites are needed: v1 returns the same token IDs and the same API as 0.23. Upgrading the data-preparation pipeline comes down to changing the version.
Where the speed comes from. Hugging Face assembled the gain from four techniques. The main one is called Bitcannon and replaces regex splitting with a SIMD bitstream for five pattern families, including GPT-2, cl100k, and o200k. Three more techniques add a per-thread cache of recurring pre-tokens, an allocation-free merge loop on preallocated buffers, and parallelism without a shared lock.
Eight workers deliver 76% of linear scaling: threads share one tokenizer without lining up in a queue.
How to install it. v1 is currently released only as a pre-release. In Rust, install it with `cargo add tokenizers --pre`, while the no-training, encoding-only variant is built with `--no-default-features --features http`.
PyPI received 1.0.0rc1 and 1.0.0rc2 on 21.09.2026; 0.23.2 from 03.09.2026 remains stable. Without the `--pre` flag, pip installs the old version.
All benchmarks were run on the Rust core. Python bindings wrap the same code, but add per-call overhead that these figures do not include.
Python bindings are expected to be simplified by the final 1.0.0, and the gains will reach transformers only after that.
