GigaToken: ~1000x faster Language model tokenization

TL;DR Highlight

Rewriting the tokenizer with SIMD and caching made it up to 1,299x faster than HF tokenizers

Who Should Read

A data or ML infrastructure developer whose preprocessing of hundreds of GB of text for training is too slow

Core Mechanics

24.53 GB/s on GPT-2
- Measured on owt_train.txt (11.9 GB) with an AMD EPYC 9565 (144 cores).
- In the same setup HF tokenizers hit 24.8 MB/s and tiktoken 36.0 MB/s.
SIMD replaces the regex engine
- Pretokenization (the first split into word-like chunks), usually left to a regex engine, is hand-written with SIMD.
- Branching is minimized and each pretoken's encoding is looked up from a cache.
Speedup varies a lot by tokenizer
- On Apple M4 Max, OLMo 2/3 is 1,299x vs HF and GPT-2 is 1,268x.
- Gemma 1 gets 17x on the same chip and 13x on the Ryzen 9800X3D.
Compatibility mode is slower than the headline
- Matching HF and tiktoken output exactly costs performance.
- The README says it is still much faster across the board.
Top speed needs the dedicated API

Community Reactions

Tokenization is under 0.1% of inference time, one commenter noted
- They said the gain is small for inference.
- Others said it should help workloads that only tokenize.
Offline training data prep is the real use, per commenters
- Tokenizing terabyte-scale corpora saves time and possibly money.
- It speeds up the loop of adjusting datasets.
Caching and regex replacement seen as reusable ideas
- One commenter cloned the repo after seeing it in the tokenization Discord.
- They said the community wants to learn from these techniques.
Requests for per-core numbers and compatibility-mode benchmarks
- One commenter wanted the README to focus on per-core performance.
- Another asked whether compatibility mode has its own benchmarks.

How to Apply

Check your tokenizer with the bench command
- Download the sample file today and run uvx gigatoken bench with your HF model name.
- Turn on --validate to confirm the output matches HF.
Swap in compatibility mode
- Wrap your HF or tiktoken tokenizer with gt.Tokenizer(...).as_hf() or .as_tiktoken().
- Compare encode_batch output to the old one, then time it.
Move bulk preprocessing to encode_files
- In pipelines handling multi-GB text files, use TextFileSource and encode_files.
- Set the separator to your document boundary token (e.g. <|endoftext|>).
Check the limits before adopting
- Check whether you use SentencePiece, WordPiece, or CJK-heavy data.
- If so, benchmark on your own data first and then decide.

Try It

copy & run

# Paste into a terminal (shell) — check GigaToken speed and correctness via uvx, no install

wget https://huggingface.co/datasets/stanford-cs336/owt-sample/resolve/main/owt_train.txt.gz
gunzip owt_train.txt.gz

uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt --validate --doc-separator "<|endoftext|>"

# Next: replace 'openai-community/gpt2' with the HF model name you use
# See options: uvx gigatoken bench --help

Terminology

TokenizerA translator that turns text into the numeric tokens a model reads. It runs in front of the model every time.

PretokenizationThe step before BPE that splits text into word-like chunks. It is usually done with a regex.

SIMDA CPU feature that processes many data items with one instruction. Like opening several checkout lanes at once.

BPEA method that merges frequent character pairs into tokens. Most LLM tokenizers such as GPT-2's use it.

Pretoken cacheA notepad of already-seen words and their tokens for reuse. It is hard to manage because word frequencies have a long tail.