Starstruk Logo MarkStarstruk

September-2024

unslothai/unslothSeptember-2024Sep 23, 2024by danielhanchen

AI Summary

Adds Qwen 2.5 support and fixes critical EOS token issues for base models.

Key Highlights

  • Qwen 2.5 support (all sizes up to 72B)
  • Fixes for Qwen 2.5 EOS token (should be `<|endoftext|>`)
  • Fixes for Qwen 2.5 chat templates (should not exist for base models)
  • 4bit quants uploaded to HuggingFace

New Features

  • Qwen 2.5 support
  • Phi 3.5 support
  • DPO fixes
  • Gemma faster inference

Full Release Notes

# Qwen 2.5 Support is here!

There are some issues with Qwen 2.5 models which Unsloth has fixed!
- Kaggle Base model finetuning notebook: https://www.kaggle.com/code/danielhanchen/kaggle-qwen-2-5-unsloth-notebook/notebook
- Kaggle Instruct model finetuning notebook: https://www.kaggle.com/code/danielhanchen/kaggle-qwen-2-5-conversational-unsloth
- Colab finetuning notebook: https://colab.research.google.com/drive/1Kose-ucXO1IBaZq5BvbwWieuubP7hxvQ?usp=sharing
- Colab conversational notebook: https://colab.research.google.com/drive/1qN1CEalC70EO1wGKhNxs1go1W9So61R5?usp=sharing

## EOS token issues
Qwen 2.5 Base models (0.5b all the way until 72b) - EOS token should be <|endoftext|> not <|im_end|>. The base models <|im_end|> is actually untrained, so it'll cause NaN gradients if you use it. You should re-pull the tokenizer from source, or you can download fixed base models from https://huggingface.co/unsloth if that helps.

## Chat template issues
- Qwen 2.5 Base models should NOT have a chat_template, this will actually cause errors especially in Unsloth's finetuning notebooks, since I check if untrained tokens exist in the chat template to counteract NaN gradients.
- Do NOT use Qwen 2.5's chat template for the base models. This will cause NaN gradients!

## 4bit uploaded models
[Qwen 2.5 0.5b](https://huggingface.co/unsloth/Qwen2.5-0.5B) 	[4bit 0.5b](https://huggingface.co/unsloth/Qwen2.5-0.5B-bnb-4bit) 	[Instruct 0.5b](https://huggingface.co/unsloth/Qwen2.5-0.5B-Instruct) 	[4bit Instruct 0.5b](https://huggingface.co/unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit)
[Qwen 2.5 1.5b](https://huggingface.co/unsloth/Qwen2.5-1.5B) 	[4bit 1.5b](https://huggingface.co/unsloth/Qwen2.5-1.5B-bnb-4bit) 	[Instruct 1.5b](https://huggingface.co/unsloth/Qwen2.5-1.5B-Instruct) 	[4bit Instruct 1.5b](https://huggingface.co/unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit)
[Qwen 2.5 3b](https://huggingface.co/unsloth/Qwen2.5-3B) 	[4bit 3b](https://huggingface.co/unsloth/Qwen2.5-3B-bnb-4bit) 	[Instruct 3b](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct) 	[4bit Instruct 3b](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct-bnb-4bit)
[Qwen 2.5 7b](https://huggingface.co/unsloth/Qwen2.5-7B) 	[4bit 7b](https://huggingface.co/unsloth/Qwen2.5-7B-bnb-4bit-bnb-4bit) 	[Instruct 7b](https://huggingface.co/unsloth/Qwen2.5-7B-Instruct) 	[4bit Instruct 7b](https://huggingface.co/unsloth/Qwen2.5-7B-Instruct-bnb-4bit)
[Qwen 2.5 14b](https://huggingface.co/unsloth/Qwen2.5-14B) 	[4bit 14b](https://huggingface.co/unsloth/Qwen2.5-14B-bnb-4bit) 	[Instruct 14b](https://huggingface.co/unsloth/Qwen2.5-14B-Instruct) 	[4bit Instruct 14b](https://huggingface.co/unsloth/Qwen2.5-14B-Instruct-bnb-4bit)
[Qwen 2.5 32b](https://huggingface.co/unsloth/Qwen2.5-32B) 	[4bit 32b](https://huggingface.co/unsloth/Qwen2.5-32B-bnb-4bit) 	[Instruct 32b](https://huggingface.co/unsloth/Qwen2.5-32B-Instruct) 	[4bit Instruct 32b](https://huggingface.co/unsloth/Qwen2.5-32B-Instruct-bnb-4bit)
[Qwen 2.5 72b](https://huggingface.co/unsloth/Qwen2.5-72B) 	[4bit 72b](https://huggingface.co/unsloth/Qwen2.5-72B-bnb-4bit) 	[Instruct 72b](https://huggingface.co/unsloth/Qwen2.5-72B-Instruct) 	[4bit Instruct 72b](https://huggingface.co/unsloth/Qwen2.5-72B-Instruct-bnb-4bit)

## What's Changed
* Phi 3.5 by @danielhanchen in https://github.com/unslothai/unsloth/pull/940
* Phi 3.5 by @danielhanchen in https://github.com/unslothai/unsloth/pull/941
* Fix DPO by @danielhanchen in https://github.com/unslothai/unsloth/pull/947
* Phi 3.5 bug fix by @danielhanchen in https://github.com/unslothai/unsloth/pull/955
* Cohere, Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/984
* Gemma faster inference by @danielhanchen in https://github.com/unslothai/unsloth/pull/987
* Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/1004
* Update README.md by @danielhanchen in https://github.com/unslothai/unsloth/pull/1033
* Update README.md by @danielhanchen in https://github.com/unslothai/unsloth/pull/1036
* fix: chat_templates.py bug by @NazimHAli in https://github.com/unslothai/unsloth/pull/1048

## New Contributors
* @NazimHAli made their first contribution in https://github.com/unslothai/unsloth/pull/1048

**Full Changelog**: https://github.com/unslothai/unsloth/compare/August-2024...September-2024