tokenizers-moonbit mirrors the HuggingFace tokenizers API closely. This guide maps common Python usage to MoonBit. The same tokenizer.json works in both.

Chinese version: docs/zh/migration-from-hf.md

Loading

HuggingFace (Python)	MoonBit
Tokenizer.from_file("tokenizer.json")	@tokenizer.from_file("tokenizer.json")
Tokenizer.from_str(s)	@tokenizer.Tokenizer::from_str(s)
Tokenizer.from_pretrained(id, local_files_only=True)	@tokenizer.from_pretrained(id) or @tokenizer.from_pretrained_cached(id, cache_dir=...)
Tokenizer.from_pretrained(id)	@hub.from_pretrained(id) on native/js, or host fetch + @tokenizer.from_pretrained_downloaded(id, json)
tok.save("dir/tokenizer.json") / directory workflows	tok.save(path) (pretty by default) or tok.save_pretrained(dir)
BPE.read_file(vocab, merges) / BPE.from_file(...) / model artifacts	@model.Model::bpe_read_file(vocab, merges) / bpe_from_file(...), or the explicit from_bpe_files loader; WordPiece/WordLevel provide matching *_read_file / *_from_file aliases

from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")

let tok = @tokenizer.from_file("tokenizer.json")

from_str takes JSON text and does no file IO, so it works on every backend. from_file uses moonbitlang/x/fs. Core @tokenizer.from_pretrained is the all-backend offline loader: it can load a local directory/file or resolve an existing HF Hub cache snapshot via $HUGGINGFACE_HUB_CACHE, $HF_HUB_CACHE, $HF_HOME/hub, or $HOME/.cache/huggingface/hub. Online download is provided by the optional @hub package on native/js: it fetches tokenizer.json, writes the same cache layout, and then reuses the core loader. It uses HuggingFace/tokenizers-like request headers on native, discovers auth tokens from HF_TOKEN, HF_TOKEN_PATH or $HF_HOME/token, and supports mirror endpoints via HF_ENDPOINT or HubDownloadOptions::new(endpoint="https://hf-mirror.com"); truthy HF_HUB_OFFLINE defaults hub options to local-only mode. Wasm/wasm-gc callers can fetch JSON in the host environment and call @tokenizer.from_pretrained_downloaded. save_pretrained(dir) writes dir/tokenizer.json, so saved artifacts can be loaded back with from_pretrained(dir).

Encoding

HuggingFace (Python)	MoonBit
tok.encode(text)	tok.encode(text)
tok.encode(text, add_special_tokens=False)	tok.encode(text, add_special_tokens=false)
tok.encode(a, b)	tok.encode_pair(a, b)
tok.encode_batch([(a, b), ...])	tok.encode_pair_batch([(a, b), ...])
tok.encode(words, is_pretokenized=True)	tok.encode_pretokenized(words)
tok.encode_batch([words, ...], is_pretokenized=True)	tok.encode_pretokenized_batch([words, ...])
tok.encode_batch([(words_a, words_b), ...], is_pretokenized=True)	tok.encode_pretokenized_pair_batch([(words_a, words_b), ...])

add_special_tokens=false / False only suppresses special tokens injected by the configured post-processor. Non-special post-processing effects still run, including Template/BERT type_ids, pair sequence_ids, and ByteLevel/RoBERTa offset trimming.

Pre-tokenized migration keeps HF's added-token extraction semantics: a special or added token embedded inside an input word is still split out before the ordinary span is passed to the model, while the configured pre-tokenizer itself is skipped.

enc = tok.encode("Hello world")
enc.ids            # [15496, 995]
enc.tokens         # ['Hello', 'Ġworld']
enc.attention_mask # [1, 1]

let enc = tok.encode("Hello world")
enc.ids            // [15496, 995]
enc.tokens         // ["Hello", "Ġworld"]
enc.attention_mask // [1, 1]

Pre-tokenized APIs skip the tokenizer's pre-tokenizer but still apply normalization, model tokenization, post-processing, truncation and padding. Offsets are measured against normalized words joined by one ASCII space.

Encoding fields

HuggingFace	MoonBit	Notes
enc.ids	enc.ids	Array[Int]
enc.tokens	enc.tokens	Array[String]
enc.type_ids	enc.type_ids	Array[Int]
enc.attention_mask	enc.attention_mask	Array[Int]
enc.special_tokens_mask	enc.special_tokens_mask	Array[Int]
enc.offsets	enc.offsets	Array[(Int, Int)]; currently char offsets

Decoding

HuggingFace (Python)	MoonBit
tok.decode(ids)	tok.decode(ids)
tok.decode(ids, skip_special_tokens=True)	tok.decode(ids, skip_special_tokens=true)
tok.decode_batch(batch)	tok.decode_batch(batch)
s = tok.decode_stream(False); s.step(id)	let (s2, chunk) = s.step(id) after tok.decode_stream(skip_special_tokens=false)

MoonBit DecodeStream::step returns the updated stream explicitly instead of mutating in place, so rebind the stream before feeding the next id.

Vocabulary lookups

HuggingFace (Python)	MoonBit
tok.token_to_id("[CLS]")	tok.token_to_id("[CLS]") → Int?
tok.id_to_token(101)	tok.id_to_token(101) → String?
tok.get_vocab_size()	tok.get_vocab_size()

Programmatic construction and added tokens

HuggingFace (Python)	MoonBit
Tokenizer(model)	@tokenizer.Tokenizer::new(model)
tok.normalizer = normalizer	tok.with_normalizer(Some(normalizer))
tok.pre_tokenizer = pre_tokenizer	tok.with_pre_tokenizer(Some(pre_tokenizer))
tok.model = model	tok.with_model(model)
tok.post_processor = processor	tok.with_post_processor(Some(processor))
tok.decoder = decoder	tok.with_decoder(Some(decoder))
tok.normalizer / tok.model / ...	tok.get_normalizer() / tok.get_model() / ...
AddedToken("<x>", single_word=True)	AddedToken::new("<x>", single_word=true)
tok.add_tokens([...])	tok.add_tokens_with_count([...]) or tok.add_tokens([...])
tok.add_special_tokens([...])	tok.add_special_tokens_with_count([...]) or tok.add_special_tokens([...])
tok.encode_special_tokens = True	tok.set_encode_special_tokens(true)
tok.encode_special_tokens	tok.get_encode_special_tokens()
tok.get_added_tokens_decoder()	tok.get_added_tokens_decoder()
tok.num_special_tokens_to_add(is_pair)	tok.num_special_tokens_to_add(is_pair=...)
tok.post_process(enc, pair, add_special_tokens=True)	tok.post_process(enc, pair=Some(pair), add_special_tokens=true)

MoonBit builders return updated tokenizer values, so rebind or chain them:

let (tok, count) = @tokenizer.Tokenizer::new(model)
  .with_pre_tokenizer(Some(@pretokenizer.PreTokenizer::whitespace_split()))
  .add_tokens_with_count([
    @tokenizer.AddedToken::new("<tag>").with_single_word(true),
  ])

The *_with_count variants return HF-style duplicate-aware counts. Ordinary added tokens keep special_tokens_mask=0; add_special_tokens registers tokens as non-normalized special entries and sets mask 1 when they are emitted. If encode_special_tokens is enabled, input occurrences of those special token strings stay on the ordinary model path and therefore receive mask 0, matching HF's switch for templating/chat use cases.

Differences to be aware of

• Booleans: MoonBit uses true/false and named arguments use name=value.
• Optionals: lookups return Int? / String? (Some/None).
• Offsets: HuggingFace returns byte offsets by default; this port currently returns char offsets.
• Configuration style: HuggingFace mutates via enable_truncation / enable_padding; MoonBit uses chainable with_truncation / with_padding plus TruncationParams::with_* and PaddingParams::with_* builders for strategy, direction, stride, pad_type_id, and pad_to_multiple_of.
• Training: deterministic WordLevel training is supported, including custom pre-tokenizers, pre-tokenized token streams, min_frequency, special_tokens, vocab_size, and HF-style frequency/lexical vocab ordering. WordPiece, BPE, and Unigram trainer MVPs are available for the same input modes with common controls such as continuation-prefix, end-of-word suffix, max_input_chars_per_word, and byte_fallback; WordPiece/BPE also support initial_alphabet, limit_alphabet, and max_token_length, plus a byte_level_alphabet() helper matching HF ByteLevel.alphabet() workflows. Trainer defaults for vocab_size, min_frequency, BPE unk_token, and Unigram unk_token now follow HF defaults; pass explicit unk_token=Some(...) / unk_piece=Some(...) or historical thresholds when you need the older MoonBit behavior.
• Regex components: common HF Split/Replace regex families are handled by shared deterministic fast paths across wasm/js/native: \s, \d, \w, ASCII classes, Unicode letter/number/punctuation/symbol classes, anchored runs, exact {2..4} / minimum {2,}/{3,}/{4,} quantifiers across the same positive and inverse class families, {1,n} bounded runs, and {2,3} / {2,4} / {3,4} ranged runs. Arbitrary complex regexes are still out of scope; Split rejects unknown patterns explicitly, while Replace falls back to literal replacement.

Verified models

With optional fixtures present, output is checked token-for-token against Python tokenizers for 39 real models, covering BPE, byte-level BPE, byte_fallback BPE, WordPiece, Unigram, WordLevel, Qwen/Llama-3/o200k Split patterns, coder, multimodal, ModernBERT-style encoder and embedding tokenizers. See PROGRESS.md(Open in new window).

Edit this page

Contributors: howtomakeaname