# Quicktok: A faster tokenizer **URL:** \ **Category:** Show and Tell\ **Created:** [June 16, 2026, 2:14am UTC](https://discuss.huggingface.co/t/quicktok-a-faster-tokenizer/176845 "2026-06-16T02:14:00Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![dmatth1](https://avatars.discourse-cdn.com/v4/letter/d/ee59a6/32.png) [@dmatth1](https://discuss.huggingface.co/u/dmatth1)\ **Post date:** [June 16, 2026, 2:14am UTC](https://discuss.huggingface.co/t/quicktok-a-faster-tokenizer/176845/1 "2026-06-16T02:14:01Z")
Been working on this a while! Should be useful for anyone trying to speed up their tokenization flow. **Introducing quicktok native (C++) and Python** quicktok is a fast, exact BPE tokenizer written in C++. Token ids are byte-identical to `tiktoken`, and encoding runs **2–3.5×** faster than `bpe-openai` (the fastest alternative I know of) and **4–11×** faster than `tiktoken` itself. It’s the fastest exact CPU tokenizer I’ve measured for these encodings. It ships cl100k, o200k, GPT-OSS (o200k\_harmony), Llama-3, and Qwen2.5/3, all byte-exact, plus bring-your-own Llama-4. This is useful for anyone doing large amounts of CPU-bound data processing — search indexing, ingesting corpora, token counting/billing — and can significantly reduce the time and cost of data ingestion. It can also be used for online request serving, such as CPU-bound inference paths (token counting, embedding serving). I’m releasing it as a Python library (`pip install quicktok-v1`) and it’s available via C++ source. Repo: [GitHub - dmatth1/quicktok: Fast exact BPE tokenizer. Byte-identical to tiktoken, 7x faster · GitHub](https://github.com/dmatth1/quicktok) ## Benchmarks Measured on 3 public corpora on my Apple M1, single thread, MB/s. Every encoder’s output was verified token-for-token identical against `tiktoken` before timing. **cl100k\_base** (GPT-3.5 / GPT-4) | encoder | The Pile | GitHub code | Common Crawl | | --- | --- | --- | --- | | **quicktok (native)** | **121.7** | **139.2** | **71.3** | | **quicktok (Python)** | **77.9** | **83.6** | **49.7** | | bpe-openai | 36.6 | 36.3 | 28.9 | | rs-bpe | 30.9 | 34.7 | 23.5 | | tiktoken-rs | 15.4 | 13.8 | 13.3 | | tiktoken (Python) | 13.6 | 12.8 | 12.3 | | TokenDagger | 11.1 | 11.9 | 10.7 | **o200k\_base** (GPT-4o) | encoder | The Pile | GitHub code | Common Crawl | | --- | --- | --- | --- | | **quicktok (native)** | **102.2** | **129.5** | **52.9** | | **quicktok (Python)** | **65.7** | **79.9** | **43.3** | | bpe-openai | 35.6 | 41.4 | 28.4 | | rs-bpe | 30.4 | 33.9 | 23.2 | | tiktoken-rs | 23.4 | 22.3 | 16.6 | | tiktoken (Python) | 20.4 | 18.6 | 14.9 | | TokenDagger | 10.4 | 11.3 | 9.3 | **quicktok** also beats llama.cpp’s tokenizer on the Llama-3 vocab by **~14×**. The parallel `encode_batch` reaches 706 MB/s native on 8 cores; from Python it sustains 550 MB/s — **24×** `tiktoken`’s batch API. The speedups hold on other architectures like x86. To keep the comparison fair, each encoder is called through the same raw API its own benchmark uses. (TokenDagger’s README claims 2–4× over tiktoken, but that’s on Llama-4/Mistral vocabs on AMD EPYC; on cl100k/o200k it lands around Python tiktoken’s level.) To reproduce run `make bench-compare` in the repo. ## How it works The fundamental algorithm is the same as `bpe-openai` (exact backtracking BPE) - see [their blog post](https://github.blog/ai-and-ml/llms/so-many-tokens-so-little-time-introducing-a-faster-more-flexible-byte-pair-tokenizer/). Much of the speedup over `bpe-openai` comes from data-structure engineering around reducing memory accesses. - **2-byte trie:** the longest-match walk reads 2 input bytes per single 8-byte slot load, with a zero-lookup direct table for CJK (Chinese/Japanese/Korean) characters. - **Dense validity memos:** merge-validity checks hit exactly-keyed caches (2 MB for 17-bit token ids, a second wide one for 200k-vocab ids; a bijective mixer means no aliasing). - **Specialized pretokenizer:** the fixed regex is compiled by hand into a scanner, no general regex. - **Single-pass product machines** — for ASCII text (most of code and English),one loop owns both the pretokenizer’s boundary rules and token emission: contractions, prefix-words, digit triples, punct runs, and the whitespace cascade are handled inline with no per-piece scanner dispatch. Only Unicode contact falls back to the general scanner. ## Caveats All comparisons are single-threaded by design - parallel/batch is available but single-threaded is fair for comparison. Multilingual text (Common Crawl) is definitely the weakest ratio. Numbers above are from an M1 and were cross-checked on x86 Xeon - the ordering holds on both but absolute MB/s moves with corpus and host. Table numbers are the native C++ build `(-march=native)`. The prebuilt PyPI wheels are portable-ABI and land lower - roughly 1.1–1.6× `bpe-openai` depending on corpus. Building from source recovers the table numbers. ## Closing notes The full methodology - corpus fetching, the exactness gate, raw-API rules — is in the bench README in the repo. If you find an input where quicktok’s ids differ from tiktoken’s that’s definitely a bug and please report it! ---
**Author:** ![masonreed11](https://avatars.discourse-cdn.com/v4/letter/m/ccd318/32.png) [@masonreed11](https://discuss.huggingface.co/u/masonreed11)\ **Post date:** [June 18, 2026, 12:45pm UTC](https://discuss.huggingface.co/t/quicktok-a-faster-tokenizer/176845/2 "2026-06-18T12:45:02Z")
Nice project. Exact `tiktoken` compatibility plus the performance gains make this genuinely useful for large-scale ingestion and token-heavy workloads. The engineering details around memory access and the specialized scanner are especially interesting. Looking forward to seeing real-world adoption numbers.