Starstruk Logo MarkStarstruk

v0.1.7

hiyouga/LlamaFactoryv0.1.7Aug 18, 2023by hiyouga

AI Summary

Introduces training script previews, checkpoint resumption, and RoPE scaling techniques for LLaMA models.

Key Highlights

  • Script preview in Web UI
  • Checkpoint resumption support
  • RoPE scaling methods (linear and NTK-aware)

New Features

  • Script preview in Web UI
  • Checkpoint resumption
  • RoPE scaling support
  • ChatGLM2-6B training
  • bfloat16 PPO training

Full Release Notes

### New features

- Preview training script in Web UI by @codemayq in #479 #511 
- Support resuming from checkpoints by @niuba in #434 (`transformers>=4.31.0` required)
- Two RoPE scaling methods: linear and NTK-aware scaling for LLaMA models (`transformers>=4.31.0` required)
- Support training the ChatGLM2-6B model
- Support PPO training in bfloat16 data type #551 

### Bug fix

- Unusual output of quantized models #278 #391 
- Runtime error in distributed DPO training #480 
- Unexpected truncation in generation #532 
- Dataset streaming error in pre-training #548 #549 
- Tensor shape mismatch in PPO training using ChatGLM2 #527 #528 
- #475 #476 #478 #481 #494 #551