Launches
Hugging Face says Tokenizers v1 is up to 30x faster
The release candidate keeps token IDs compatible while rebuilding the hot path around SIMD splitting, reusable buffers and native parallelism.
3-30x encode speed Source-backed 2 sources