v0.5.3

hiyouga/LlamaFactoryv0.5.3Feb 28, 2024by hiyouga

AI Summary

Adds support for DoRA and QLoRA for AWQ/AQLM quantized models, alongside Gemma model support.

Key Highlights

Support for DoRA (Weight-Decomposed LoRA)
2-bit QLoRA support for AWQ/AQLM models
Support for Google Gemma models

New Features

DoRA algorithm support
2-bit QLoRA for AWQ/AQLM
Flash-attn package for Windows
PPO trainer fixes
Atom models support

Full Release Notes

### New features

- Support **[DoRA](https://arxiv.org/abs/2402.09353)** (Weight-Decomposed LoRA)
- Support QLoRA for the AWQ/AQLM quantized models, now **2-bit QLoRA** is feasible
- Provide some example scripts in https://github.com/hiyouga/LLaMA-Factory/tree/main/examples

### New models

- Base models
  - Gemma (2B/7B)
- Instruct/Chat models
  - Gemma-it (2B/7B)

### Bug fix

- Add flash-attn package for Windows user by @codemayq in #2514 
- Fix ppo trainer #1163 by @stephen-nju in #2525 
- Support atom models by @Rayrtfr in #2531 
- Support role in webui by @lungothrin in #2575 
- Bump accelerate to 0.27.2 and fix #2552 by @Katehuuh in #2608 
- Fix #2512 #2516 #2532 #2533 #2629