v0.5.0

hiyouga/LlamaFactoryv0.5.0Jan 20, 2024by hiyouga

AI Summary

Celebrates 10k stars by adding agent tuning, Unsloth integration for speed, and function calling support.

Key Highlights

Agent tuning support for tool usage
Unsloth integration for 170% faster LoRA training
Function calling in API and Web modes

New Features

Agent tuning (`--dataset glaive_toolcall`)
Function calling support
Unsloth integration (`--use_unsloth`)
MPS device support
Data engine refactor
Saving logic refactor for value heads

Full Release Notes

### Congratulations on 10k stars 🎉 Make LLM fine-tuning easier and faster together with LLaMA-Factory ✨

### New features

- Support **agent tuning** for most models, you can fine-tune any LLMs with `--dataset glaive_toolcall` for tool using #2226 
- Support function calling in both **API** and **Web** mode with fine-tuned models, same as the [OpenAI's format](https://platform.openai.com/docs/api-reference/chat/create)
- LLaMA Factory 🤝 [Unsloth](https://github.com/unslothai/unsloth), enjoy **170%** LoRA training speed with `--use_unsloth`, see benchmarking [here](https://github.com/hiyouga/LLaMA-Factory/wiki/Performance-comparison)
- Supports fine-tuning models on MPS device #2090 

### New models

- Base models
  - Phi-2 (2.7B)
  - InternLM2 (7B/20B)
  - SOLAR-10.7B
  - DeepseekMoE-16B-Base
  - XVERSE-65B-2
- Instruct/Chat models
  - InternLM2-Chat (7B/20B)
  - SOLAR-10.7B-Instruct
  - DeepseekMoE-16B-Chat
  - Yuan (2B/51B/102B)

### New datasets

- Supervised fine-tuning datasets
  - deepctrl dataset
  - Glaive function calling dataset v2

### Core updates

- Refactor data engine: clearer dataset alignment, easier templating and tool formatting
- Refactor saving logic for models with value head #1789 
- Use ruff code formatter for stylish code

### Bug fix

- Bump transformers version to 4.36.2 by @ShaneTian in #1932 
- Fix requirements by @dasdristanta13 in #2117 
- Add Machine-Mindset project by @JessyTsui in #2163 
- Fix typo in readme file by @junuMoon in #2194 
- Support resize token embeddings with ZeRO3 by @liu-zichen in #2201 
- Fix #1073 #1462 #1617 #1735 #1742 #1789 #1821 #1875 #1895 #1900 #1908 #1907 #1909 #1923 #2014 #2067 #2081 #2090 #2098 #2125 #2127 #2147 #2161 #2164 #2183 #2195 #2249 #2260