b10003
ggml-org/llama.cppb10003Jul 14, 2026by github-actions[bot]same day as b10002
What changed since b10002
Tokenize tool refactored onto shared common_params_parse, gaining model-sourcing and tokenization CLI flags.
New features
- The tokenize example now accepts the shared model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN).
- Tokenization-specific flags --ids, --stdin, --no-bos, --no-parse-special and --show-count are now registered as common arguments.
- Tokenize can now pull models directly from Hugging Face or a local directory instead of requiring a hand-supplied path.
Improvements
- Replaced the tokenize tool's hand-rolled argv parsing with the shared common_params_parse helper for consistency across examples.
- Dropped bespoke Windows UTF-8 handling and file reading in favor of shared common helpers.
- Error reporting in the tokenize tool now goes through LOG_ERR rather than raw fprintf(stderr).
Upgrade notes
Only the tokenize example binary is affected; library and server behavior are unchanged. parse_special still defaults to true for tokenize, so existing invocations should produce identical output. Scripts that scrape tokenize error output may see slightly different formatting now that LOG_ERR is used.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
Refactored the tokenize tool to use shared argument parsing (`common_params_parse`) for consistency, exposing new model sourcing and tokenization flags.
Key Highlights
- Migrated tokenize tool to use `common_params_parse` for argument handling.
- Exposed model sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) and tokenization flags (--ids, --stdin, --no-bos, --no-parse-special, --show-count).
- Improved error handling using LOG_ERR.
Release Notes (excerpt)
<details open> tokenize : align usage by using common args (#25516) Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin, --no-bos, --no-parse-special and --show-count as common args. parse_special defaults to true for TOKENIZE to preserve the old…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system