tokenizers v1: encode, decode and scaling, measured
Tokenizers v1 is a new release of the Hugging Face tokenization library. It adds encode, decode, and scaling features, with performance metrics measured. The update improves tokenization speed and memory usage for developers.