Starstruk Logo MarkStarstruk

v0.1.7

feyninc/mandexv0.1.7Mar 23, 2026by chonknick

AI Summary

Replaces the HuggingFace `tokenizers` library with a custom `tokie` library to improve initialization speed and reduce dependency overhead.

Key Highlights

  • Replaced HuggingFace `tokenizers` with custom `tokie` library
  • Tokenizer initialization is 20-50x faster
  • Reduces transitive dependencies by ~260
  • Removes runtime dependency on HuggingFace Hub

New Features

  • Replacement of tokenizers with `tokie`

Full Release Notes

## What's Changed

- Replace HuggingFace `tokenizers` with `tokie` for reranking
  - ~20-50x faster tokenizer initialization (pre-built .tkz from CDN vs JSON parsing)
  - ~260 fewer transitive dependencies
  - 100% token-accurate, identical reranking results
  - No runtime dependency on HuggingFace Hub
- Tokenizer served from `cdn.mandex.dev/models/tokenizer.tkz` alongside reranker model