v0.5.2

hiyouga/LlamaFactoryv0.5.2Feb 20, 2024by hiyouga

AI Summary

Introduces block expansion for LLaMA Pro, Qwen1.5 models, and fixes for DeepSpeed Zero3 with MoE models.

Key Highlights

Block expansion support for LLaMA Pro
Comprehensive Qwen1.5 model support
DeepSpeed Zero3 fixes for MoE and Mixtral

New Features

Block expansion support
`use_rslora` option for LoRA
Qwen1.5 models (0.5B to 72B)
DeepSpeed Zero3 fixes for MoE
German SFT and DPO datasets

Full Release Notes

### New features

- Support **block expansion** in [LLaMA Pro](https://github.com/TencentARC/LLaMA-Pro), see `tests/llama_pro.py` for usage
- Add `use_rslora` option for the LoRA method

### New models

- Base models
  - Qwen1.5 (0.5B/1.8B/4B/7B/14B/72B)
  - DeepSeekMath-7B-Base
  - DeepSeekCoder-7B-Base-v1.5
  - Orion-14B-Base
- Instruct/Chat models
  - Qwen1.5-Chat (0.5B/1.8B/4B/7B/14B/72B)
  - MiniCPM-2B-SFT/DPO
  - DeepSeekMath-7B-Instruct
  - DeepSeekCoder-7B-Instruct-v1.5
  - Orion-14B-Chat
  - Orion-14B-Long-Chat
  - Orion-14B-RAG-Chat
  - Orion-14B-Plugin-Chat

### New datasets

- Supervised fine-tuning datasets
  - SlimOrca (en)
  - Dolly (de)
  - Dolphin (de)
  - Airoboros (de)
- Preference datasets
  - Orca DPO (de)

### Bug fix

- Fix `torch_dtype` check in export model by @fenglui in #2262 
- Add Russian locale to LLaMA Board by @seoeaa in #2264 
- Remove manually set `use_cache` in export model by @yhyu13 in #2266 
- Fix DeepSpeed Zero3 training with MoE models by @A-Cepheus in #2283 
- Add a patch for full training of the Mixtral model using DeepSpeed Zero3 by @ftgreat in #2319 
- Fix bug in data pre-processing by @lxsyz in #2411 
- Add German sft and dpo datasets by @johannhartmann in #2423 
- Add version checking in `test_toolcall.py` by @mini-tiger in #2435 
- Enable parsing of SlimOrca dataset by @mnmueller in #2462 
- Add tags for models when pushing to hf hub by @younesbelkada in #2474 
- Fix #2189 #2268 #2282 #2320 #2338 #2376 #2388 #2394 #2397 #2404 #2412 #2420 #2421 #2436 #2438 #2471 #2481