v0.2.0

EricLBuehler/mistral.rsv0.2.0Jul 19, 2024by github-actions[bot]

AI Summary

Introduces PagedAttention (major performance gain), supports .bin/.pt/.pth extensions, and adds Starcoder 2 GGUF.

Key Highlights

  • PagedAttention support (beating llama.cpp)
  • Support for .bin, .pt, .pth extensions
  • Starcoder 2 GGUF support
  • Optimized multi-batch and inference performance

Breaking Changes

  • Version 0.2.0

New Features

  • PagedAttention
  • Multiple file extensions
  • Starcoder 2 GGUF

Full Release Notes

## New features
- Support .bin, .pt, .pth extensions
- Add Starcoder 2 GGUF
- 🔥 PagedAttention - beating llama.cpp running GGUF plus all the throughput benefits 😉 
- Optimized performance and memory usage

## Rust MSRV

MSRV of `mistral.rs` v0.2.0 is 1.75.

## What's Changed
* Fix SWA order (flip it) for Gemma 2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/554
* Support .bin, .pt, .pth extensions by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/557
* Update readme by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/558
* Fix Starcoder 2 ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/559
* Update deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/560
* Add the starcoder2 GGUF arch by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/561
* Readme update for starcoder2 gguf by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/562
* Fix PyPI release trigger by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/566
* Optimize multi-batch and inference performance with PagedAttention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/552
* [Breaking] Version 0.2.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/527
* Paged attention support for vision models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/567
* Automatically use paged attn on cuda, get memory size by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/568
* Add docs link for vision loader by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/570
* Add matching for valid model weight names by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/571
* Remove ensure about no paged attn for vision models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/573
* Add percentage utilization support to paged attn by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/574
* Include block engine in paged attn metadata by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/576
* Update deps and sync Candle by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/578
* Optimize CLIP model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/579
* Use softmax_last_dim in CLIP by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/580
* Fix method of calculating paged attn with util percent by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/581
* Handle windows in paged attn build by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/577
* Warn instead of error when paged attn not supported by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/583
* Warn instead of error when paged attn for adapters not supported by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/584
* Add support for lm_head to adapter models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/586
* Add default plotly feature by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/587
* Improve memory handling of PagedAttention with GGUF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/590
* Fix Windows build on cuda w/ PagedAttention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/589
* Update cuda kernels build.rs on windows by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/591
* Bump version to 0.2.0 and update docs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/582


**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.1.26...v0.2.0

## Install mistralrs-server 0.2.0

### Install prebuilt binaries via shell script

```sh
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-installer.sh | sh
```

## Download mistralrs-server 0.2.0

|  File  | Platform | Checksum |
|--------|----------|----------|
| [mistralrs-server-aarch64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-aarch64-apple-darwin.tar.xz) | Apple Silicon macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-aarch64-apple-darwin.tar.xz.sha256) |
| [mistralrs-server-x86_64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-apple-darwin.tar.xz) | Intel macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-apple-darwin.tar.xz.sha256) |
| [mistralrs-server-x86_64-unknown-linux-gnu.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz) | x64 Linux | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz.sha256) |