v0.5.0

## Highlights

After a few days of tight iteration and more than 6,000 lines of new code, we’re ready to introduce `DSQ` quantization to `deepseek-ocr.rs`.

Unlike formats like `GGUF` or plain `safetensors` that usually ship full float checkpoints, `DSQ` behaves as a relatively compact patch on top of the base weights: the original float weights stay as‑is, and `DSQ` only adds a quantized snapshot of the heavy linear layers. This lets `DeepSeek-OCR`, `PaddleOCR-VL` and future OCR/VLM models share the same snapshot pipeline via small adapters instead of inventing a new weight format every time.

We chose this weight‑patch design because the goal is to **cut VRAM/RAM usage without needlessly hurting model quality**. Concretely:

- For `DeepSeek-OCR`, the adapters target the large language Transformer linears: attention projections (e.g. `self_attn.q_proj/k_proj/v_proj/o_proj`), MLP blocks (e.g. `mlp.gate_proj/up_proj/down_proj`, including MoE `experts` and `shared_experts`), and, when enabled, the multimodal projector (`model.projector.layers.weight` / `bias`). The `lm_head.weight` is also part of the snapshot, but when the primary precision is `Q4_K` or `Q6_K`, both `lm_head.weight` and `model.projector.layers.weight` are automatically promoted to `Q8_0` to stay more conservative on the most critical heads.
- For `PaddleOCR-VL`, the adapters cover the text decoder linears (attention `q_proj/k_proj/v_proj/o_proj` and MLP `gate_proj/up_proj/down_proj`, plus `lm_head.weight`), as well as the visual side: SigLIP encoder linears under `visual.vision_model.encoder.layers.*`, the `visual.vision_model.head.*` attention and MLP blocks, and the cross‑modal bridge linears `mlp_AR.linear_1/linear_2`.
- Embeddings, layer norms, positional encodings, routing logic and other “fragile” components stay in float and are not written into `DSQ`.

On the loading side, the `DSQ` runtime batches these linear specs and supports parallel loading, which helps tame cold‑start time for larger models where startup used to be dominated by weight IO and decoding. Out of the box, this release ships three tiers — `Q4_K`, `Q6_K` and `Q8_0` — so you can pick the trade‑off that matches your hardware and tolerance for compression.


## New Quantized Models Table

The following quantized `model-id`s are new in this release (the original `deepseek-ocr` / `paddleocr-vl` stay available):

| Model ID           | Base Model       | Precision | Suggested Use Case                        |
|--------------------|------------------|-----------|-------------------------------------------|
| `deepseek-ocr-q4k`   | `deepseek-ocr`   | `Q4_K`    | Tight VRAM, local deployments, batch jobs |
| `deepseek-ocr-q6k`   | `deepseek-ocr`   | `Q6_K`    | Day‑to‑day balance of quality and size    |
| `deepseek-ocr-q8k`   | `deepseek-ocr`   | `Q8_0`    | When you want to stay close to full precision |
| `paddleocr-vl-q4k`   | `paddleocr-vl`   | `Q4_K`    | Heavily compressed doc/table deployments  |
| `paddleocr-vl-q6k`   | `paddleocr-vl`   | `Q6_K`    | Common engineering setups                 |
| `paddleocr-vl-q8k`   | `paddleocr-vl`   | `Q8_0`    | Accuracy‑leaning, still smaller than FP16 |


## Memory / VRAM Comparison

Using `FP32` as a 1.0× baseline, you can think of the different precisions roughly like this:

| Precision | Approx. Size vs FP32 | Notes                                        |
|-----------|----------------------|----------------------------------------------|
| `FP32`    | 1.0×                 | Baseline                                     |
| `FP16`    | ~0.5×                | Standard half precision                      |
| `Q8_0`    | ~0.25–0.3×           | Solid default for many workloads             |
| `Q6_K`    | ~0.2–0.25×           | Smaller again while keeping good quality     |
| `Q4_K`    | ~0.14×               | Roughly one seventh of FP32, very aggressive |

In short: lower precision → smaller weights → less pressure on VRAM/RAM, with `Q4_K` being the most aggressive tier.


## How to choose quantization type based on hardware

> This isn’t a hard rule, but it should give you a reasonable starting point:

### `DeepSeek-OCR` (~3B MoE)

| VRAM profile      | Recommended precision | Example `model-id`    | Notes                                         |
|-------------------|-----------------------|-----------------------|-----------------------------------------------|
| 8GB GPU           | `Q4_K`                | `deepseek-ocr-q4k`    | Conservative choice to stay within VRAM       |
| 10–12GB GPU       | `Q6_K`                | `deepseek-ocr-q6k`    | Good everyday default, balanced trade‑off     |
| ≥16GB GPU         | `Q8_0`                | `deepseek-ocr-q8k`    | Prefer this when you care more about quality  |

### `PaddleOCR-VL` (~0.9B Dense + SigLIP)

| VRAM profile      | Recommended precision | Example `model-id`      | Notes                                      |
|-------------------|-----------------------|-------------------------|-------------------------------------------|
| 4–6GB GPU         | `Q4_K`                | `paddleocr-vl-q4k`      | Safe choice for smaller cards             |
| 6–8GB GPU         | `Q6_K`                | `paddleocr-vl-q6k`      | Reasonable default for most deployments   |
| ≥8GB GPU          | `Q8_0`                | `paddleocr-vl-q8k`      | Use when you want extra accuracy          |

## 新特性

在经过几天的紧张开发以后，加上 6000+ 行新增代码，我们在此向你介绍全新的 `DSQ` 量化。

不同于 `GGUF` / `safetensors` 这类“整份完整权重”格式，`DSQ` 更像是对基础权重的一个相对 compact 的 `patch`：  
基础 float 权重仍然保留，而 `DSQ` 只额外存储量化后的线性层快照。后续无论是 `DeepSeek-OCR`、`PaddleOCR-VL`，还是未来接入的其他 OCR / VLM，只要写一个适配器，就可以把这个模型挂到同一套 `DSQ` 权重 patch 流水线上。

在加载阶段，我们对这些量化权重支持 **并行加载**（按线性层批量拉起），在大模型上可以明显缩短冷启动阶段的权重准备时间。

同时，为了在降低显存占用的前提下尽量不影响模型性能，我们没有简单粗暴地把所有参数一股脑量化，而是只对参数量最大但相对“耐折腾”的线性层做了 patch：

- 对于 `DeepSeek-OCR`，适配器会覆盖语言 Transformer 块里的注意力投影和 MLP：包括 `self_attn.q_proj/k_proj/v_proj/o_proj`，`mlp.gate_proj/up_proj/down_proj`，以及 MoE `experts` / `shared_experts` 中对应的线性层；在启用 projector 的配置下，还会纳入 `model.projector.layers.weight` / `bias`。`lm_head.weight` 也在快照中，但当主量化精度选择 `Q4_K` / `Q6_K` 时，会自动将 `lm_head.weight` 和 `model.projector.layers.weight` 提升到 `Q8_0`，在关键输出头上更保守。
- 对于 `PaddleOCR-VL`，适配器会覆盖文本 decoder 中的注意力投影 (`self_attn.q_proj/k_proj/v_proj/o_proj`) 与 MLP (`mlp.gate_proj/up_proj/down_proj`)，以及 `lm_head.weight`；在视觉侧，会量化 `SigLIP` encoder 的线性层（`visual.vision_model.encoder.layers.*` 中的 `q_proj/k_proj/v_proj/out_proj` 与 `fc1/fc2`），`visual.vision_model.head.*` 中的注意力与 MLP 线性层，以及跨模态 bridge 的 `mlp_AR.linear_1/linear_2`。
- 词嵌入、LayerNorm、位置编码、路由逻辑等更敏感的部分则保持 float，不写入 `DSQ` 快照。

本次更新支持了 `Q4_K`、`Q6_K`、`Q8_0` 三种精度，覆盖从“极致省显存”到“尽量贴近全精度”的不同使用场景。


## 新增量化模型 Table

本次新增的量化模型 ID 如下（原始的 `deepseek-ocr` / `paddleocr-vl` 依然保留）：

| 模型 ID           | 基础模型      | 精度   | 推荐场景                             |
|-------------------|---------------|--------|--------------------------------------|
| `deepseek-ocr-q4k`  | `deepseek-ocr`    | `Q4_K` | 显存紧张、本地部署、批量任务        |
| `deepseek-ocr-q6k`  | `deepseek-ocr`    | `Q6_K` | 质量 / 体积均衡的日常使用           |
| `deepseek-ocr-q8k`  | `deepseek-ocr`    | `Q8_0` | 希望尽量接近全精度的场景           |
| `paddleocr-vl-q4k`  | `paddleocr-vl`    | `Q4_K` | 文档 / 表格场景下的高压缩部署       |
| `paddleocr-vl-q6k`  | `paddleocr-vl`    | `Q6_K` | 工程类应用的一般默认配置            |
| `paddleocr-vl-q8k`  | `paddleocr-vl`    | `Q8_0` | 更关心精度、又想比 FP16 更省的场景  |

从用户视角，使用方式不变：在 CLI / 配置 / HTTP 请求里，把 `model` 换成上表中的任意一个 ID 即可。


## 内存 / 显存占用对比

以 `FP32 = 1.0×` 基准粗略对比不同精度的体积：

| 精度    | 相对 FP32 体积（约） | 说明                                   |
|---------|-----------------------|----------------------------------------|
| `FP32`  | 1.0×                  | 基准线                                 |
| `FP16`  | ~0.5×                 | 常规半精度                             |
| `Q8_0`  | ~0.25–0.3×            | 适合作为通用量化配置                  |
| `Q6_K`  | ~0.2–0.25×            | 在更小体积下维持不错的质量            |
| `Q4_K`  | ~0.14×                | 大致是 FP32 的七分之一左右，极致压缩  |

整体来说，精度越低，权重体积越小，显存 / 内存压力越低；其中 `Q4_K` 是压缩最激进的一档。


## 如何根据硬件选择量化精度

> 仅作「起步参考」，实际能否跑得动还会受 batch 大小、分辨率、后处理等影响。

### `DeepSeek-OCR`（~3B MoE）

| 显存情况（大致） | 推荐量化精度 | 示例 `model-id`        | 说明                             |
|------------------|--------------|------------------------|----------------------------------|
| 8GB GPU          | `Q4_K`       | `deepseek-ocr-q4k`     | 尽量保守，优先确保能稳定跑起来   |
| 10–12GB GPU      | `Q6_K`       | `deepseek-ocr-q6k`     | 日常默认选择，质量 / 体积比较均衡 |
| ≥16GB GPU        | `Q8_0`       | `deepseek-ocr-q8k`     | 更在意输出质量时可以优先考虑    |

### `PaddleOCR-VL`（~0.9B Dense + SigLIP）

| 显存情况（大致） | 推荐量化精度 | 示例 `model-id`        | 说明                                      |
|------------------|--------------|------------------------|-------------------------------------------|
| 4–6GB GPU        | `Q4_K`       | `paddleocr-vl-q4k`     | 小卡场景下的稳妥选项                      |
| 6–8GB GPU        | `Q6_K`       | `paddleocr-vl-q6k`     | 一般本地部署、服务端默认都可以从这档起步 |
| ≥8GB GPU         | `Q8_0`       | `paddleocr-vl-q8k`     | 更在意精度时可以直接用 `Q8_0`            |



**Full Changelog**: https://github.com/TimmyOVO/deepseek-ocr.rs/compare/v0.4.1...v0.5.0