Starstruk Logo MarkStarstruk

v0.2.5

EricLBuehler/mistral.rsv0.2.5Aug 16, 2024by github-actions[bot]

AI Summary

Improves ISQ parsing, implements prompt chunking, and adds GPTQ and HQQ quantization support.

Key Highlights

  • Refactored ISQ quant parsing
  • Prompt chunking implementation
  • GPTQ quantization support
  • HQQ quantization support

New Features

  • Prompt chunking
  • GPTQ quantization
  • HQQ quantization
  • Improved ISQ parsing

Full Release Notes

## What's Changed
* Refactor ISQ quant parsing by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/664
* Refactor server examples to use OpenAI Python client by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/665
* Implement prompt chunking by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/623
* Python example and server example cleanup by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/668
* Implement GPTQ quantization by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/467
* Update deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/672
* Rework the automatic dtype selection feature by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/676
* Fix backend Candle fork Metal, flash attn, also Llama linear by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/681
* Use converted tokenizer.json in tests by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/682
* Refactor ISQ and mistralrs-quant by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/683
* Fix metal build for isq by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/686
* Add missing error case in automatic dtype selection feature by @ac3xx in https://github.com/EricLBuehler/mistral.rs/pull/685
* fix null in tool type response by @wseaton in https://github.com/EricLBuehler/mistral.rs/pull/687
* Implement HQQ quantization by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/677
* Bump version to 0.2.5 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/688

## New Contributors
* @ac3xx made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/685
* @wseaton made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/687

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.2.4...v0.2.5
## Install mistralrs-server 0.2.5

### Install prebuilt binaries via shell script

```sh
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-installer.sh | sh
```

## Download mistralrs-server 0.2.5

|  File  | Platform | Checksum |
|--------|----------|----------|
| [mistralrs-server-aarch64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-aarch64-apple-darwin.tar.xz) | Apple Silicon macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-aarch64-apple-darwin.tar.xz.sha256) |
| [mistralrs-server-x86_64-apple-darwin.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-x86_64-apple-darwin.tar.xz) | Intel macOS | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-x86_64-apple-darwin.tar.xz.sha256) |
| [mistralrs-server-x86_64-unknown-linux-gnu.tar.xz](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz) | x64 Linux | [checksum](https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-x86_64-unknown-linux-gnu.tar.xz.sha256) |