v0.2.0
EricLBuehler/mistral.rsv0.2.0Jul 19, 2024by github-actions[bot]
AI Summary
Introduces PagedAttention (major performance gain), supports .bin/.pt/.pth extensions, and adds Starcoder 2 GGUF.
Key Highlights
- PagedAttention support (beating llama.cpp)
- Support for .bin, .pt, .pth extensions
- Starcoder 2 GGUF support
- Optimized multi-batch and inference performance
Breaking Changes
- Version 0.2.0
New Features
- PagedAttention
- Multiple file extensions
- Starcoder 2 GGUF
Full Release Notes
## New features - Support .bin, .pt, .pth extensions - Add Starcoder 2 GGUF - 🔥 PagedAttention - beating llama.cpp running GGUF plus all the throughput benefits 😉 - Optimized performance and memory usage ## Rust MSRV MSRV of `mistral.rs` v0.2.0 is 1.75. ## What's Changed * Fix SWA order (flip it) for Gemma 2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/554 * Support .bin, .pt, .pth extensions by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/557 * Update readme by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/558 * Fix Starcoder 2 ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/559 * Update deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/560 * Add the starcoder2 GGUF arch by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/561 * Readme update for starcoder2 gguf by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/562 * Fix PyPI release trigger by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/566 * Optimize multi-batch and inference performance with PagedAttention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/552 * [Breaking] Version 0.2.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/527 * Paged attention support for vision models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/567 * Automatically use paged attn on cuda, get memory size by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/568 * Add docs link for vision loader by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/570 * Add matching for valid model weight names by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/571 * Remove ensure about no paged attn for vision models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/573 * Add percentage utilization support to paged attn by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/574 * Include block engine in paged attn metadata by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/576 * Update deps and sync Candle by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/578 * Optimize CLIP model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/579 * Use softmax_last_dim in CLIP by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/580 * Fix method of calculating paged attn with util percent by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/581 * Handle windows in paged attn build by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/577 * Warn instead of error when paged attn not supported by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/583 * Warn instead of error when paged attn for adapters not supported by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/584 * Add support for lm_head to adapter models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/586 * Add default plotly feature by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/587 * Improve memory handling of PagedAttention with GGUF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/590 * Fix Windows build on cuda w/ PagedAttention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/589 * Update cuda kernels build.rs on windows by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/591 * Bump version to 0.2.0 and update docs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/582 **Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.1.26...v0.2.0 ## Install mistralrs-server 0.2.0 ### Install prebuilt binaries via shell script ```sh curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-installer.sh | sh ``` ## Download mistralrs-server 0.2.0 | File | Platform | Checksum | |--------|----------|----------| | [mistralrs-server-aarch64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-aarch64-apple-darwin.tar.xz) | Apple Silicon macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-aarch64-apple-darwin.tar.xz.sha256) | | [mistralrs-server-x86_64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-apple-darwin.tar.xz) | Intel macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-apple-darwin.tar.xz.sha256) | | [mistralrs-server-x86_64-unknown-linux-gnu.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz) | x64 Linux | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz.sha256) |