v0.1.7

asheshgoplani/agent-deckv0.1.7Mar 23, 2026by chonknick

AI Summary

Replaced the HuggingFace tokenizers library with a custom `tokie` library to significantly speed up tokenizer initialization and reduce dependency overhead.

Key Highlights

~20-50x faster tokenizer initialization
~260 fewer transitive dependencies
100% token-accurate, identical reranking results
No runtime dependency on HuggingFace Hub
Tokenizer served from CDN

New Features

Custom tokenizer library (`tokie`)
CDN-hosted tokenizer files

Full Release Notes

## What's Changed

- Replace HuggingFace `tokenizers` with `tokie` for reranking
  - ~20-50x faster tokenizer initialization (pre-built .tkz from CDN vs JSON parsing)
  - ~260 fewer transitive dependencies
  - 100% token-accurate, identical reranking results
  - No runtime dependency on HuggingFace Hub
- Tokenizer served from `cdn.mandex.dev/models/tokenizer.tkz` alongside reranker model