Starstruk Logo MarkStarstruk

v3.0.0

karakeep-app/karakeepv3.0.0Jun 30, 2026by lyogavin

AI Summary

This major version update enables the execution of large open-source models on small GPUs through native FP8 support and ensures compatibility with the latest Hugging Face libraries and model families.

Key Highlights

  • Run massive models like DeepSeek-V3 (671B) and Llama 3.1 (405B) on minimal hardware (e.g., 4GB or 8GB GPUs)
  • Native FP8 support for block-FP8 checkpoints including Qwen3 and DeepSeek-V3
  • Expanded support for the latest models including Qwen3 (MoE), Phi-4, and Mixtral-8x7B
  • Full compatibility with modern Hugging Face transformers and accelerate versions

New Features

  • Native FP8 support for block-FP8 models
  • Support for Qwen3 (dense and MoE variants)
  • Support for DeepSeek-V3 and DeepSeek-V2-Lite
  • Support for Phi-4 and Mixtral-8x7B
  • Reworked layer streaming for better compatibility with Transformers
  • Runtime precision follows model native dtype (e.g., bfloat16) instead of forcing fp16

Full Release Notes

## AirLLM v3.0.0

Big update: AirLLM now runs today's largest open models on tiny GPUs, with full support for the latest model families and Hugging Face versions — still no quantization, distillation, or pruning required.

### Highlights
- **Run the biggest open models on a single small GPU.** Stream 70B models on 4GB, 405B Llama 3.1 on 8GB, and even **DeepSeek-V3 (671B) on ~12GB**.
- **Native FP8 support.** Pre-quantized FP8 (block-FP8) checkpoints now load and run correctly — including DeepSeek-V3 and the Qwen3-FP8 family.
- **Latest models supported**, including Qwen3 (dense + MoE, e.g. Qwen3-32B, Qwen3-30B-A3B, Qwen3-235B-A22B-FP8), DeepSeek-V3, Phi-4, Mixtral-8x7B, and DeepSeek-V2-Lite.
- **Up to date with modern Hugging Face.** Works with current `transformers` / `accelerate` releases, so a plain `pip install airllm` just works — no manual dependency juggling.

### Improvements & fixes
- Reworked layer streaming to build on the standard Transformers model path for better model compatibility and `generate()` behavior.
- Runtime precision now follows each model's native dtype (e.g. bfloat16) instead of being forced to fp16, fixing garbled output on very deep models.
- Fixed weight loading for layers whose tensors span multiple checkpoint shards (affected large FP8/MoE models).
- More robust shard naming and attention-implementation fallback.

### Install / upgrade
```bash
pip install --upgrade airllm
```

See the [README](https://github.com/lyogavin/airllm#readme) for quickstart and the full list of supported models.