b10003

ggml-org/llama.cppb10003Jul 14, 2026by github-actions[bot]same day as b10002

What changed since b10002

low risk

Tokenize tool refactored onto shared common_params_parse, gaining model-sourcing and tokenization CLI flags.

New features

  • The tokenize example now accepts the shared model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN).
  • Tokenization-specific flags --ids, --stdin, --no-bos, --no-parse-special and --show-count are now registered as common arguments.
  • Tokenize can now pull models directly from Hugging Face or a local directory instead of requiring a hand-supplied path.

Improvements

  • Replaced the tokenize tool's hand-rolled argv parsing with the shared common_params_parse helper for consistency across examples.
  • Dropped bespoke Windows UTF-8 handling and file reading in favor of shared common helpers.
  • Error reporting in the tokenize tool now goes through LOG_ERR rather than raw fprintf(stderr).

Upgrade notes

Only the tokenize example binary is affected; library and server behavior are unchanged. parse_special still defaults to true for tokenize, so existing invocations should produce identical output. Scripts that scrape tokenize error output may see slightly different formatting now that LOG_ERR is used.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

Refactored the tokenize tool to use shared argument parsing (`common_params_parse`) for consistency, exposing new model sourcing and tokenization flags.

Key Highlights

  • Migrated tokenize tool to use `common_params_parse` for argument handling.
  • Exposed model sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) and tokenization flags (--ids, --stdin, --no-bos, --no-parse-special, --show-count).
  • Improved error handling using LOG_ERR.

Release Notes (excerpt)

<details open> tokenize : align usage by using common args (#25516) Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin, --no-bos, --no-parse-special and --show-count as common args. parse_special defaults to true for TOKENIZE to preserve the old…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories