tokenizers-moonbit

HuggingFace-compatible tokenizers for MoonBit.

Load tokenizer.json, encode and decode with one pure MoonBit runtime across wasm, wasm-gc, js and native.

Built for portable tokenizer parity

One pure MoonBit implementation targets wasm, wasm-gc, js and native without native addons.

Load standard HuggingFace tokenizer.json files and keep the familiar pipeline semantics.

Optional fixture tests compare against Python tokenizers across classic and modern model families.