v0.1.8
cheaper-inference/OmniRoutev0.1.8Mar 23, 2026by chonknick
Upgrading?
AI Summary
Overhauled reranking performance by 3.3x by implementing a single pass reranking strategy, using pre-optimized ORT models, and enabling zero-copy memory-mapped file access with multi-threaded inference.
Key Highlights
- Single rerank pass across all packages instead of per-package reranking
- Pre-optimized ORT format for faster deserialization
- Zero-copy memory-mapped file access (mmap)
- Multi-threaded inference using half available cores
New Features
- Single rerank pass optimization
- ORT format model support
- Multi-threaded inference (capped at 4 cores)
Full Release Notes
## What's Changed ### Reranking Performance Overhaul (3.3x faster) - **Single rerank pass** — collect all FTS5 results across packages, rerank once instead of per-package (eliminates redundant ONNX session loads) - **ORT format** — pre-optimized flatbuffer model (`reranker.ort`) for faster deserialization, with automatic fallback to `.onnx` - **mmap model loading** — zero-copy memory-mapped file access via `commit_from_memory_directly` - **Multi-threaded inference** — uses half available cores (capped at 4) for ONNX intra-op parallelism ### Benchmarks | Scenario | v0.1.7 | v0.1.8 | Speedup | |----------|--------|--------|---------| | All packages (11) | 1.01s | **0.31s** | 3.3x | | Single package | 0.09s | **0.04s** | 2.3x |
Related Repositories
openclaw/openclaw★ 392,786
The AI that really does things. Any OS. Any Platform. The lobster way. 🦞
nilbuild/developer-roadmap★ 370,160
Interactive roadmaps, guides and other educational content to help developers grow in their careers.
morluto/rea★ 365,972
Reverse engineer anything with agents, from app behavior down to native binaries.
anomalyco/opencode★ 220,197
The open source coding agent.
n8n-io/n8n★ 208,350
Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.