v0.1.7
Friedrich-M/UniMatev0.1.7Mar 23, 2026by chonknick
Upgrading?
AI Summary
Replaces HuggingFace `tokenizers` with a custom `tokie` library for faster initialization and reduced dependencies, serving the tokenizer from a CDN.
Key Highlights
- Replaced HuggingFace `tokenizers` with `tokie`
- 20-50x faster tokenizer initialization
- Reduced transitive dependencies by ~260
- No runtime dependency on HuggingFace Hub
New Features
- Replacement of tokenizer library with `tokie`
Full Release Notes
## What's Changed - Replace HuggingFace `tokenizers` with `tokie` for reranking - ~20-50x faster tokenizer initialization (pre-built .tkz from CDN vs JSON parsing) - ~260 fewer transitive dependencies - 100% token-accurate, identical reranking results - No runtime dependency on HuggingFace Hub - Tokenizer served from `cdn.mandex.dev/models/tokenizer.tkz` alongside reranker model
Related Repositories
public-apis/public-apis★ 491,256
A collective list of free APIs
NousResearch/hermes-agent★ 256,127
The agent that grows with you
yt-dlp/yt-dlp★ 200,304
A feature-rich command-line audio/video downloader
microsoft/markitdown★ 191,593
Python tool for converting files and office documents to Markdown.
Significant-Gravitas/AutoGPT★ 187,703
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.