v0.2.1

hiyouga/LlamaFactoryv0.2.1Nov 9, 2023by hiyouga

AI Summary

Adds NEFTune support, a wide variety of new model variants, and dataset loading improvements including ShareGPT formats and caching.

Key Highlights

Support for NEFTune trick in supervised fine-tuning
Expanded model support including Yi, Mistral, Falcon, and Deepseek families
Enhanced dataset support with ShareGPT formats and pre-processing caching

New Features

NEFTune integration
ShareGPT dataset format loading
Multiple response generation in demo API
Dataset file caching via cache_path argument
push_to_hub support
LLaMA Board improvements

Full Release Notes

### New features

- Support [**NEFTune**](https://arxiv.org/abs/2310.05914) trick for supervised fine-tuning by @anvie in #1252 
- Support loading dataset in the sharegpt format - read [data/readme](https://github.com/hiyouga/LLaMA-Factory/blob/main/data/README.md) for details
- Support generating multiple responses in demo API via the `n` parameter
- Support caching the pre-processed dataset files via the `cache_path` argument
- Better LLaMA Board (pagination, controls, etc.)
- Support `push_to_hub` argument #1088 

### New models

- Base models
  - ChatGLM3-6B-Base
  - Yi (6B/34B)
  - Mistral-7B
  - BlueLM-7B-Base
  - Skywork-13B-Base
  - XVERSE-65B
  - Falcon-180B
  - Deepseek-Coder-Base (1.3B/6.7B/33B)
- Instruct/Chat models
  - ChatGLM3-6B
  - Mistral-7B-Instruct
  - BlueLM-7B-Chat
  - Zephyr-7B
  - OpenChat-3.5
  - Yayi (7B/13B)
  - Deepseek-Coder-Instruct (1.3B/6.7B/33B)

### New datasets

- Pre-training datasets
  - RedPajama V2
  - Pile
- Supervised fine-tuning datasets
  - OpenPlatypus
  - ShareGPT Hyperfiltered
  - ShareGPT4
  - UltraChat 200k
  - AgentInstruct
  - LMSYS Chat 1M
  - Evol Instruct V2

### Bug fix

- Fix full-parameter DPO training #1383 #1422 (inspired by @mengban )
- Fix tokenizer config by @lvzii in #1436 
- Fix #1197 #1215 #1217 #1218 #1228 #1232 #1285 #1287 #1290 #1316 #1325 #1349 #1356 #1365 #1411 #1418 #1438 #1439 #1446