v0.0.1

anomalyco/opentuiv0.0.1Jun 27, 2026by chonknick

AI Summary

First public release of pulpie, a fast and CPU-friendly HTML content extraction tool using encoder models. It achieves high fidelity to MinerU-HTML with verified byte-parity and ROUGE-5 scores.

Key Highlights

Achieves MinerU-HTML parity with 65/65 byte-parity tests and identical ROUGE-5 F1 scores (0.8625)
Three model variants (Small, Base, Large) available for different quality and cost trade-offs
Clean HTML or Markdown output from raw HTML
CPU-friendly architecture using the Orange encoder family (210M - 2.1B parameters)
Verified end-to-end performance on WebMainBench English (6,647 pages)

New Features

Extractor class with automatic model download on first use
Markdown output support via pip install pulpie[markdown]
Integration of Orange encoder models for block classification
WebMainBench benchmark verification

Full Release Notes

First public release of **pulpie** — fast, CPU-friendly web content extraction from HTML using encoder models.

## Install

```bash
pip install pulpie            # core
pip install pulpie[markdown]  # + markdown output
```

## What it does

pulpie extracts the main content from raw HTML and returns clean HTML or markdown. It classifies HTML blocks with small encoder models (the **Orange** family), reaching autoregressive-extractor quality at a fraction of the cost.

```python
from pulpie import Extractor

extractor = Extractor()             # downloads pulpie-orange-small (210M) on first use
result = extractor.extract(html)
print(result.markdown)              # clean markdown
print(result.html)                  # clean HTML
print(result.n_main, result.n_other)
```

## MinerU-HTML parity

pulpie's `simplify`/`reconstruct` faithfully reproduce MinerU-HTML's `simplify_html` / `map_to_main` (Apache-2.0), the segmentation format the Orange models were distilled on. Verified two ways:

- **Offline byte-parity:** 65/65 tests reproduce MinerU's exact output across 13 real-page fixtures.
- **End-to-end ROUGE-5** on full WebMainBench English (6,647 pages, orange-small):

  | Simplification | ROUGE-5 F1 |
  |----------------|-----------|
  | MinerU-HTML (reference) | 0.8625 |
  | **pulpie (this release)** | **0.8625** |

  Byte-faithful end-to-end (Δ vs MinerU = +0.0000), a +0.14 improvement over the prior simplification.

## Model zoo

| Name | HF repo | Size | Notes |
|------|---------|------|-------|
| Orange Small | `chonkie-ai/pulpie-orange-small` | 210M | Best value; default |
| Orange Base | `chonkie-ai/pulpie-orange-base` | 610M | Distilled |
| Orange Large | `chonkie-ai/pulpie-orange-large` | 2.1B | Highest quality |

## Credits

Built on MinerU-HTML / Dripper (Ma et al., 2025) — their `simplify_html` preprocessing, block-annotation scheme, and the WebMainBench benchmark are foundational. Thank you for releasing your tools and data.