Starstruk Logo MarkStarstruk

v0.1.8

EricLBuehler/mistral.rsv0.1.8May 16, 2024by EricLBuehler

AI Summary

Major update featuring low-precision matmul kernels, a new loading API, and improved CTRL-C handling.

Key Highlights

  • Low-precision matmul via f16 kernels
  • New loading API for local filesystem
  • Better handling of CTRL-C in interactive mode
  • LoRA support improvements

New Features

  • Low-precision matmul kernels
  • New loading API
  • Improved CTRL-C handling

Full Release Notes

## Overview
- Documentation improvements
- Better handling of CTRL-C in interactive mode
- Matmul via low-precision kernels to take advantage of faster cuBLAS GEMM kernels (thanks @lucasavila00)
- New loading API (thanks @Jeadie)
- Various small bug fixes
- Reduce dependancy complexity (thanks @LLukas22)

## What's Changed
* bug fix: llama kv cache part by @keisuke-niimi-insightedge-jp in https://github.com/EricLBuehler/mistral.rs/pull/300
* Refactor cache manager and kv cache by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/304
* Update the docs for ISQ and misc by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/310
* Make `pyo3` an optional dependency in `mistralrs-core` by @LLukas22 in https://github.com/EricLBuehler/mistral.rs/pull/303
* Update kv cache by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/312
* Print gguf metadata consistently by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/313
* Allow loading LoRA without activating adapters and fix bugs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/306
* Remove spurious tokenizer warnings by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/314
* Better handling of ctrlc by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/315
* Add analysis bot by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/316
* Quantized: Use cublas for prompt by @lucasavila00 in https://github.com/EricLBuehler/mistral.rs/pull/238
* Support loading model into pipeline from local filesystem by @Jeadie in https://github.com/EricLBuehler/mistral.rs/pull/308
* Fix the ctrlc handler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/318
* Don't force QLlama to have >2 input dims  @Jeadie by @Jeadie in https://github.com/EricLBuehler/mistral.rs/pull/320
* Matmul via f16 when possible by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/317

## New Contributors
* @keisuke-niimi-insightedge-jp made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/300
* @Jeadie made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/308

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.1.7...v0.1.8