This page lists implemented tokenizer.json components. Unknown component types raise UnsupportedComponent at load time.
Chinese version: docs/zh/components.md
type | Status | Notes |
|---|
BPE / byte-level BPE | ✅ | byte_fallback, fuse_unk, ignore_merges supported |
WordPiece | ✅ | greedy longest-match, ## prefix, end_of_word_suffix, max_input_chars_per_word |
Unigram | ✅ | Viterbi; byte_fallback and fuse_unk supported |
WordLevel | ✅ | whole-token lookup + unk |
type | Status |
|---|
Lowercase, Strip, Replace, Prepend, Sequence | ✅ |
StripAccents | ✅ (NFD-minus-Mn table) |
BertNormalizer (clean_text / handle_chinese_chars / strip_accents / lowercase) | ✅ |
NFC / NFD / NFKC / NFKD | ✅ Unicode table + decomposition/recomposition |
Nmt | ✅ Moses/NMT cleanup for SentencePiece-era tokenizers |
ByteLevel normalizer | ✅ UTF-8 bytes mapped to the GPT-2 byte alphabet |
Precompiled (SentencePiece charsmap) | ✅ Parses SentencePiece binary double-array precompiled_charsmap; empty maps keep NFKC + full Unicode whitespace folding with ASCII fast path |
type | Status |
|---|
ByteLevel (GPT-2 scanner) | ✅ |
Whitespace, WhitespaceSplit, BertPreTokenizer, Punctuation, Metaspace, Sequence | ✅ |
Split with GPT-2 / Qwen-Llama3 / o200k / CLIP / CJK / digit-triplet / trailing-whitespace regex families | ✅ |
Digits, Delimiter, FixedLength, UnicodeScripts | ✅ |
Split regex strategy | ✅ supported deterministic families load normally; unsupported JSON regex patterns raise UnsupportedComponent at load time (manually constructed runtime fallback remains one piece) |
type | Status |
|---|
TemplateProcessing, BertProcessing, RobertaProcessing, ByteLevel, Sequence | ✅ |
type | Status |
|---|
ByteLevel, WordPiece, BPEDecoder, Metaspace, Fuse, Replace, Strip, Sequence | ✅ (WordPiece cleanup uses a single-pass BERT punctuation/contraction fast path) |
ByteFallback | ✅ |
CTC | ✅ |
With optional fixtures present, encode matches Python tokenizers token-for-token for 39 real models: gpt2, roberta, llama, bert/bert-cased, distilbert, t5, albert, xlm-roberta, Qwen2.5, Qwen3, DeepSeek-V2, Phi-3, Mistral, Falcon, StarCoder2, GPT-NeoX, CLIP, DeBERTa, Llama-3.2, Phi-4-mini, DeepSeek-R1-Distill-Qwen, DeepSeek-V3.2, GPT-OSS, GLM-4.5, Granite-4, Qwen3-Coder, Qwen3-VL, BGE-M3, multilingual-E5, ModernBERT, GTE-ModernBERT, MiniLM, BGE-large-en, Jina embeddings, Nomic Embed, E5-small, MixedBread and SmolLM2.
- Regex strategy: this project deliberately does not embed a full backtracking or fully general Unicode regex engine. The supported
Split / Replace regex families are deterministic scanners chosen from real HuggingFace tokenizer configurations. Regex constructs such as look-ahead/look-behind, backreferences, arbitrary alternation/grouping semantics, and unsupported Unicode property combinations fail explicitly instead of being approximated. - Precompiled charsmap clusters: the charsmap transform mirrors HF 0.22.2: grapheme clusters (a base character plus trailing combining marks) shorter than 6 UTF-8 bytes are looked up in the trie as a whole first, so decomposed
e + U+0301 composes to é on t5-style normalizers; misses fall back to per-character lookups, and 6+ byte clusters (Hangul jamo pairs) stay uncomposed, matching the upstream guard. - Precompiled charsmap: tokenizer.json
precompiled_charsmap payloads are base64-decoded into the SentencePiece double-array trie and applied per Unicode scalar; empty/null maps retain the common SPM NFKC + Unicode whitespace folding path with an ASCII fast path. - Arbitrary
Split regex: well-known GPT-2 / Qwen-Llama3 / o200k / CLIP / CJK / digit-triplet patterns plus common simple spans such as literal and escaped-literal alternatives (foo|bar, (foo) / (?:foo), (foo|bar), (?:foo|bar), foo|a\\.b), {2,}, including anchored and word-boundary forms like ^foo$ / \\bfoo\\b / ^(?:foo|bar) / (?:foo|bar)$ / \\b(?:foo|bar)\\b / ^\\b(?:foo|bar)\\b$, \s+, \S+, ^\s+, \s+$, \s{2,} / \s{3,} / \s{4,} and exact \s{2} / \s{3} / \s{4}, [\r\n]+ with {2,} / {3,} / {4,} and exact {2} / {3} / {4} forms, [^\S\r\n]+-style horizontal whitespace ([^\S\r\n]+ plus {2,} / {3,} / {4,} and exact {2} / {3} / {4} aliases), \d+, \D+, bracketed digit aliases ([\d]+, [^\d]+, \P{N}+), min digit runs (\d{2,} / [\d]{2,}), exact digit runs (\d{2} / \d{3} / \d{4}), bounded/ranged digit runs (\d{1,2} / \d{1,3} / \d{1,4} / \d{2,4} and \p{N} aliases), anchored digit/word/letter runs, word/letter quantifier runs (\w{2,} / \w{2} / {1,2} / {1,3} / {1,4} bounded plus {2,3} / {2,4} / {3,4} ranged word, ASCII letter/alnum and Unicode letter forms / [A-Za-z]{2,} / \p{L}{3}), punctuation/symbol quantifier runs (\p{P}{2,} / \p{P}{2} / \p{S}{2} / {1,n} bounded and {2,n} ranged punctuation, symbol and \p{P}\p{S} union forms / [\p{P}\p{S}]{2,}), inverse exact/min/bounded/ranged runs (\D{3,}, \D{4}, \S{2,}, \W{2}, \P{L}{3,}, \P{P}{4}, \P{S}{2,}, inverse ASCII classes, [^\p{P}\p{S}]{3,} and [^\s\p{L}\p{N}]{4}), \w+, \W+, ASCII alnum/letter classes ([A-Za-z0-9]+, [A-Za-z]+ and inverse forms), \p{L}+, \P{L}+, punctuation classes (\p{P}+ / \P{P}+) and symbol classes (\p{S}+ / \P{S}+) are recognized, including common union/inverse classes such as [\p{P}\p{S}]+ and [^\s\p{L}\p{N}]+. Unsupported regexes raise UnsupportedComponent at load time instead of silently producing mismatched splits. A full Unicode regex engine remains future work. - Regex
Replace: Replace normalizer/decoder supports common whitespace regex replacements such as \s+, anchored positive/inverse class runs such as ^\s+, \s+$, ^\S+, \D+$, ^\P{Letter}+, \P{Punctuation}+$, ^[^\p{P}\p{S}]+ and [^\s\p{L}\p{N}]+$, \s{2,} / \s{3,} / \s{4,} and exact \s{2} / \s{3} / \s{4}, [\r\n]+ with min/exact {2..4} quantifiers, [^\S\r\n]+ and [ \t]+ horizontal whitespace min/exact {2..4} forms, {2,}, digit runs \d+ / [\d]+ / \D+ / \P{N}+, min digit runs such as \d{2,} / [\d]{2,}, exact digit runs (\d{2} / \d{3} / \d{4}), word/letter quantifier runs (\w{2,} / \w{3,} / \w{2} / {1,2} / {1,3} / {1,4} bounded / {2,3} / {2,4} / {3,4} ranged word, ASCII letter/alnum and Unicode letter forms / [A-Za-z]{2,} / \p{L}{3}), punctuation/symbol quantifier runs (\p{P}{2,} / \p{P}{2} / \p{S}{2} / {1,n} bounded and {2,n} ranged punctuation, symbol and \p{P}\p{S} union forms / [\p{P}\p{S}]{2,}), common exact/min/bounded positive and inverse runs such as \d{1,3}, \D{1,3}, \D{3,}, \D{4}, \W{1,2}, \W{2}, \P{L}{1,4}, \P{L}{3,} and [^\s\p{L}\p{N}]{1,3} / {4}, anchored digit/word/letter/punctuation/symbol runs, word runs \w+ / \W+, ASCII alnum/letter runs [A-Za-z0-9]+ / [A-Za-z]+ and inverse forms, and letter/punctuation/symbol runs (\p{L}+, \p{P}+, \p{S}+ and inverse forms), plus common union/inverse classes like [\p{P}\p{S}]+ and [^\s\p{L}\p{N}]+, and simple literal alternatives such as foo|bar / ^foo$ / \\bfoo\\b / (?:foo|bar) / ^(?:foo|bar) / (?:foo|bar)$ / \\b(?:foo|bar)\\b where every branch is an escaped or plain literal. Normalizer and Decoder Replace dispatch quantified bounded/ranged forms before the generic span-building fallback, keeping those micro paths direct and faster than HF; more complex regex replacement remains future work. - Punctuation classification: BertPreTokenizer and the Punctuation pre-tokenizer use HuggingFace's classifier (
is_ascii_punctuation || is_punctuation). The P* table is generated empirically from Python tokenizers 0.22.2 itself (scripts/gen_unicode_punct.py sweeps the BertPreTokenizer classifier over planes 0-2), so script punctuation (Syriac/Tibetan/Myanmar/Thai ...), vertical/small form variants, the katakana middle dot and astral-plane P* are all covered; ASCII symbols ($ + < = > ^ \ | ~`) count as punctuation via the ASCII union. CJK ideographs are no longer split per character by BertPreTokenizer (that is BertNormalizer's `handle_chinese_chars` concern), matching HF. `merged_with_previous`/`merged_with_next` merge only the first/last delimiter of a run, matching HF 0.22.2. - Offsets (original-referential): encode offsets are char offsets into the ORIGINAL text, mirroring HuggingFace's NormalizedString alignment semantics: normalization keeps a per-normalized-char original span (NFD marks collapse onto the base char, NFC composition keeps the first composed-from char's span, Strip/Prepend/Replace/BertNormalizer/charsmap each map their edits back), added-vocabulary segmentation and model-token emission convert through the alignment column, and ByteLevel/Metaspace piece rewrites (multi-byte expansion, space->▁, prefix inserts) map back to input char coordinates. Verified by the behavioral sweep (synthetic 857 -> 971/1008; real models 59 -> 94/100).
- Offsets: char-based by default, relative to the original text. Optional byte-offset encode APIs are available for HuggingFace-style byte offsets.
- Decoder semantics: BPEDecoder substring-replaces every suffix occurrence per token — with a space in all tokens except the last, where it is removed; Metaspace decode drops ALL replacement chars in the first token for
always/first prepend schemes (never maps them to spaces); CTC collapses consecutive duplicates, deletes the pad token string anywhere (including inside tokens), and with cleanup=true applies the wordpiece cleanup per token before mapping the word delimiter to a space (it stays literal otherwise); WordPiece and CTC cleanup use the exact upstream rule list; Replace decoders are per-token replace-all and patterns outside the supported deterministic regex families fail explicitly (both the standalone and Sequence paths) instead of silently replacing literally. - Truncation overflow windows: encode mirrors HF
tokenizers 0.22.2: windows land in enc.overflowing for both directions and stride = 0 / stride > 0, post-processors wrap each window, and fixed padding pads windows to the same target. Pair encode builds the 0.22.x window cross-product (per-side windows combined with the opposite mains and windows, main pair excluded); note that HuggingFace main-branch (0.23-dev) has since replaced that scheme with plain per-side truncation windows. Configuration-time stride validation matches upstream: with_truncation / enable_truncation raise when stride > 0 and the stride exceeds max_length - num_special_tokens_to_add(single); loading a tokenizer.json stays permissive and an invalid stored config surfaces at encode time, as upstream. - Batching: single-threaded by design for wasm/js targets.
- Performance: BPE merging uses a priority-queue heap with lazy stale removal plus word caching; WordPiece and Unigram also cache repeated words; BPE loading fills dense reverse vocab tables directly; repeated/alternating
from_str loads reuse a small parsed-JSON cache while constructing fresh tokenizer state each time. - Training: deterministic WordLevel training is supported with default
WhitespaceSplit, caller-provided pre-tokenizers, or pre-tokenized token streams, including min_frequency, special_tokens, vocab_size, and HF-style frequency/lexical vocab ordering. Deterministic WordPiece, BPE, and Unigram trainer MVPs are also available with the same input modes and common knobs such as continuation-prefix / end-of-word suffix / max_input_chars_per_word / byte_fallback; WordPiece and BPE training additionally support HF-style initial_alphabet / limit_alphabet, WordPiece also preserves end_of_word_suffix through encode and JSON round-trip, WordPiece/BPE support max_token_length, and byte_level_alphabet() exposes the 256 ByteLevel symbols for GPT-2/RoBERTa style training. - Constructed tokenizer serialization: trained WordLevel, WordPiece, BPE, and Unigram tokenizers serialize common pre-tokenizers such as ByteLevel, Metaspace, Punctuation, Split, Digits, Delimiter, FixedLength, UnicodeScripts, and Sequence.