Gigatoken: ~1000x faster Language model tokenization
Summary
Gigatoken is a Rust-based tokenizer claimed to be ~1000x faster than HuggingFace tokenizers, enabling GB/s throughput. The readme covers installation, compatibility with HF tokenizers and tiktoken, and extensive benchmarks across various CPUs, highlighting SIMD and caching optimizations.