October-2024

We fixed a gradient accumulation bug which was actually discovered since 2021 [here](https://github.com/huggingface/transformers/issues/14638), and rediscovered [here](https://github.com/huggingface/trl/issues/2175). Read more in our blog post: https://unsloth.ai/blog/gradient

We have a [Colab Notebook](https://colab.research.google.com/drive/1z0XJU2FCzDC8oyXa2Nd4jCxylRMI-o0-?usp=sharing) for Llama 3.2 using the fixed trainer and a [Kaggle Notebook](https://www.kaggle.com/code/danielhanchen/fixed-kaggle-llama-3-2-1b-3b-conversation) as well.

Essentially theoretically `bsz * ga` should be equivalent to full batch training with no gradient accumulation, but weirdly the training losses do no match up:
![](https://unsloth.ai/cgi/image/Before_-_Standard_gradient_accumulation_UQOFkUggudXuV9dzrh8MA.svg?width=3840&quality=80&format=auto)

We fixed it in Unsloth!
![](https://unsloth.ai/cgi/image/After_-_Unsloth_gradient_accumulation_6Y4pJdJF0vruzradUpymY.svg?width=3840&quality=80&format=auto)

To use Unsloth's fixed trainer with gradient accumulation, use:
```python
from unsloth import unsloth_train
# trainer_stats = trainer.train() << Buggy if using gradient accumulation
trainer_stats = unsloth_train(trainer) # << Fixed gradient accumulation
```

Please update Unsloth on local machines (no need for Colab / Kaggle) via:
```bash
pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
```
Read our blog post: https://unsloth.ai/blog/gradient for more details!

## What's Changed
* Llama 3.2 by @danielhanchen in https://github.com/unslothai/unsloth/pull/1058
* Fix merges by @danielhanchen in https://github.com/unslothai/unsloth/pull/1079
* Handle absolute paths for save_to_gguf using pathlib by @giuliabaldini in https://github.com/unslothai/unsloth/pull/1120
* Only remove folder in sentencepiece check if it was created by @giuliabaldini in https://github.com/unslothai/unsloth/pull/1121
* Gradient Accumulation Fix by @danielhanchen in https://github.com/unslothai/unsloth/pull/1134

## New Contributors
* @giuliabaldini made their first contribution in https://github.com/unslothai/unsloth/pull/1120

**Full Changelog**: https://github.com/unslothai/unsloth/compare/September-2024...October-2024