Starstruk Logo MarkStarstruk

v3.3.0

lyogavin/airllmv3.3.0Aug 28, 2026by lyogavin

AI Summary

This release focuses on optimizing the inference of the Qwen3.8-Flash-Next model, enabling it to run on a single GPU with minimal VRAM usage through advanced streaming techniques.

Key Highlights

  • Qwen3.8-Flash-Next runs on a single RTX 4090 in 5.95GB VRAM.
  • Utilizes file-mapped n-gram tables and decoder layer streaming.
  • Requires in-tree `qwen4_exp` from the Transformers main branch.
  • Needs ~360GB disk space or the `delete_original=True` option to reclaim space.

New Features

  • Streaming of Qwen3.8-Flash-Next model on single card.

Full Release Notes

Qwen3.8-Flash-Next (`Qwen/Qwen3.8-Flash-Next`) streams on a single card in **5.95GB** VRAM (RTX 4090). The ~51B n-gram table is file-mapped on the host; decoder layers stream.

```bash
pip install -U airllm
pip install git+https://github.com/huggingface/transformers.git
```

Flash-Next needs in-tree `qwen4_exp` (GitHub `transformers` main today). A 64GB RAM machine is enough; plan ~360GB disk, or pass `delete_original=True` to reclaim the originals after the split.