v0.2.0

hiyouga/LlamaFactoryv0.2.0Oct 15, 2023by hiyouga

AI Summary

Major refactor introducing LongLoRA, support for training large models (Qwen-14B, InternLM-20B), Ascend NPU support, and benchmark integration.

Key Highlights

LongLoRA support for LLaMA models
Training support for Qwen-14B and InternLM-20B
Ascend NPU support and MMLU/C-Eval/CMMLU benchmark integration

Breaking Changes

Renamed repository to LLaMA Factory
Changed argument names from max_source_length to cutoff_len

New Features

LongLoRA integration
Qwen-14B and InternLM-20B training
Web UI state recovery
Ascend NPU support
train_on_prompt option
MMLU, C-Eval, and CMMLU benchmarks

Full Release Notes

### New features

- Support [**LongLoRA**](https://github.com/dvlab-research/LongLoRA) for the LLaMA models
- Support training the Qwen-14B and InternLM-20B models
- Support training state recovery for the all-in-one Web UI
- Support Ascend NPU by @statelesshz in #975 
- Integrate MMLU, C-Eval and CMMLU benchmarks

### Modifications

- Rename repository to **LLaMA Factory** (former LLaMA Efficient Tuning)
- Use the `cutoff_len` argument instead of `max_source_length` and `max_target_length` #944 
- Add a `train_on_prompt` option #1184 

### Bug fix

- Fix numeric error caused by the layer norm dtype in https://github.com/hiyouga/LLaMA-Factory/commit/84b7486885c600e5e65c5ba9095d56ecc2502977 [1]
- Fix bugs in PPO Trainer by @mmbwf in #900 
- Fix #424 #762 #814 #887 #913 #1000 #1026 #1032 #1064 #1068 #1074 #1086 #1097 #1176 #1177 #1190 #1191 

[1] https://github.com/huggingface/transformers/pull/25598#discussion_r1335345914