Usage

About 526 wordsAbout 2 min

tokenizers-moonbit loads a HuggingFace tokenizer.json and runs encode/decode on every MoonBit backend. This guide covers the common tasks.

Chinese version: docs/zh/usage.md

Loading

// From JSON text (backend-agnostic, no file IO):
let tok = @tokenizer.Tokenizer::from_str(json_text)

// From a file path (uses moonbitlang/x/fs):
let tok = @tokenizer.from_file("tokenizer.json")

Both functions may raise @types.TokenizerError on malformed JSON or an unsupported component. Handle it with try/catch or propagate it with raise.

HuggingFace Hub

The core @tokenizer.from_pretrained stays synchronous and works on all backends. It loads local files/directories or an already-populated HF Hub cache:

let tok = @tokenizer.from_pretrained("bert-base-uncased")

For native/js applications that want online download, use the optional @hub package. It downloads tokenizer.json, writes the standard HF-style cache, then delegates parsing back to the core tokenizer package:

let tok = @hub.from_pretrained("bert-base-uncased")

let opts = @hub.HubDownloadOptions::new(
  revision="main",
  cache_dir=Some(".hf-cache"),
  endpoint="https://hf-mirror.com", // optional mirror URL for mainland China
  token=None, // or set HF_TOKEN in the environment
)
let tok2 = @hub.from_pretrained("org/model", options=opts)

The optional Hub downloader is supported on native and js via moonbitlang/async/http. Its native requests use a HuggingFace/tokenizers-like User-Agent plus standard Accept/Authorization headers; browser/JS builds rely on the runtime-supplied User-Agent because fetch forbids setting it manually. On wasm/wasm-gc, fetch JSON in the host application and call @tokenizer.from_pretrained_downloaded(model_id, json_text, cache_dir=...) or Tokenizer::from_str directly.

Encoding

let enc = tok.encode("Hello world")
// enc.ids                 : Array[Int]
// enc.tokens              : Array[String]
// enc.type_ids            : Array[Int]
// enc.attention_mask      : Array[Int]
// enc.special_tokens_mask : Array[Int]
// enc.offsets             : Array[(Int, Int)]   // char offsets

add_special_tokens defaults to true and controls whether the post-processor template runs. Special tokens that literally appear in the text are still recognized.

let enc = tok.encode("text", add_special_tokens=false)

Pairs

let enc = tok.encode_pair("question", "context")
// builds e.g. [CLS] question [SEP] context [SEP] with type_ids 0/1

Batches

let batch = tok.encode_batch(["first text", "second"])

Batching is single-threaded by design. Combine it with padding for rectangular outputs.

Decoding

let text = tok.decode(enc.ids)                          // skips special tokens
let raw  = tok.decode(enc.ids, skip_special_tokens=false)

Truncation

let tok = @tokenizer.Tokenizer::from_str(json)
  .with_truncation(Some(@tokenizer.TruncationParams::new(128)))
let enc = tok.encode(long_text)   // capped to 128 tokens

TruncationParams fields: max_length, stride, direction (Left/Right). With stride > 0, trimmed tail windows are placed in enc.overflowing.

Padding

// Fixed length:
let tok = @tokenizer.Tokenizer::from_str(json)
  .with_padding(Some(@tokenizer.PaddingParams::new(Fixed(64))))

// Pad each batch to the longest member:
let tok = @tokenizer.Tokenizer::from_str(json)
  .with_padding(Some(@tokenizer.PaddingParams::new(BatchLongest)))
let batch = tok.encode_batch(texts)   // all rows same length

PaddingParams::new(strategy, pad_id?, pad_token?); further fields: direction (Left/Right), pad_type_id, pad_to_multiple_of.

Padded positions get attention_mask = 0 and special_tokens_mask = 1.

Vocabulary lookups

tok.token_to_id("[CLS]")   // Int?
tok.id_to_token(101)       // String?
tok.get_vocab_size()       // Int

Errors

fn load(json : String) -> Unit {
  match (try? @tokenizer.Tokenizer::from_str(json)) {
    Ok(tok) => ...
    Err(@types.ParseError(msg)) => println("bad json: \{msg}")
    Err(@types.UnsupportedComponent(c)) => println("unsupported: \{c}")
    Err(_) => ()
  }
}